Add Character Encoding to Heroku buildpack

Using charlock homes library in your heroku application

In this tutorial, we’ll add the support for the charlock_holmes library to a heroku build pack. The icu4c is a native extension that is used for processing character encoding in ruby.

Setup: Take a look at [setting up a custom build pack](/how-do-i/ruby/setup-custom-ruby-buildpack.html) article.

First, create an S3 bucket to store the binary library for use in your build pack. Let’s call it my-heroku-binaries. In the lib/language_pack/base.rb, add a variable for your binary bucket.

MY_BINARY_URL = 'https://s3.amazonaws.com/my-heroku-binaries'

Build the binary for ICU4C using vulcan. See Ryan Daigle’s tutorial on building with vulcan. Download the binary and put it in your bucket.

Building via vulcan:

vulcan build -v -s icu/source -p /app/vendor/icu4c-49.1.2 \
  -c "./configure --prefix=/app/vendor/icu4c-49.1.2 --disable-samples --disable-tests && make && make install"

In the compile method in lib/language_pack/ruby.rb add a call to the install_icu4c method. The library will eventually be installed into /app/vendor/icu4c-49.1.2.

def compile
    Dir.chdir(build_path)
    remove_vendor_bundle
    install_ruby
    install_jvm
    setup_language_pack_environment
    setup_profiled
    install_icu4c
    allow_git do
      install_language_pack_gems
      build_bundler
      create_database_yml
      install_binaries
      run_assets_precompile_rake_task
    end
    super
  end

Also update the setup_language_pack_environment and the setup_profiled methods in the lib/language_pack/ruby.rb to reference the libraries for compilation by updating the LD_LIBRARY_PATH.

 # sets up the environment variables for the build process
  def setup_language_pack_environment
    setup_ruby_install_env

    config_vars = default_config_vars.each do |key, value|
      ENV[key] ||= value
    end
    ENV["GEM_HOME"] = slug_vendor_base
    ENV["PATH"]     = "#{ruby_install_binstub_path}:#{slug_vendor_base}/bin:#{config_vars["PATH"]}"
    ENV["LD_LIBRARY_PATH"] = "vendor/#{ICU4C_VENDOR_PATH}/lib"
  end

  # sets up the profile.d script for this buildpack
  def setup_profiled
    set_env_override "GEM_PATH", "$HOME/#{slug_vendor_base}:$GEM_PATH"
    set_env_default  "LANG",     "en_US.UTF-8"
    set_env_override "PATH",     "$HOME/bin:$HOME/#{slug_vendor_base}/bin:$PATH"
    set_env_default  "LD_LIBRARY_PATH", "/app/vendor/#{ICU4C_VENDOR_PATH}/lib"

    if ruby_version_jruby?
      set_env_default "JAVA_OPTS", default_java_opts
      set_env_default "JRUBY_OPTS", default_jruby_opts
      set_env_default "JAVA_TOOL_OPTIONS", default_java_tool_options
    end
  end

Update the build_bundler and change the env_vars within the Dir.mktmpdir("libyaml-") do |tmpdir| loop to include CHARLOCK_HOMES reference to icu lib:

env_vars       = "env BUNDLE_GEMFILE=#{pwd}/Gemfile BUNDLE_CONFIG=#{pwd}/.bundle/config BUNDLE_BUILD__CHARLOCK_HOLMES=\"--with-icu-dir=#{pwd}/vendor/#{ICU4C_VENDOR_PATH}\" CPATH=#{yaml_include}:$CPATH CPPATH=#{yaml_include}:$CPPATH LIBRARY_PATH=#{yaml_lib}:$LIBRARY_PATH RUBYOPT=\"#{syck_hack}\""

Create the method below:

def install_icu4c
  dir = File.join('vendor', ICU4C_VENDOR_PATH)
  FileUtils.mkdir_p dir
  Dir.chdir(dir) do
    run("curl #{MY_BINARY_URL}/icu4c-#{ICU4C_VERSION}.tar.gz -s -o - | tar xzf -")
  end
end

Add charlock_holmes to your Gemfile and you are all good to go.