Skip to content
Snippets Groups Projects
  1. May 14, 2017
    • Jungshik Shin's avatar
      Update ICU to 59.1 · 87232d8d
      Jungshik Shin authored
      * Highlights:
        - Emoji 5.0 data (partial; Emoji_Component property not included)
        - CLDR 31.0.1 (http://blog.unicode.org/2017/03/cldr-version-31-released.html)
          UTC and GMT are treated as distinct)
        - New case mapping API for styled text
        - C++ 11 is required
        - char16_t for UChar (UTF-16)
        - Source code is in UTF-8
      
      * Size changes
      
        common: 10,130,560 => 10,175,056
        android: 6,573,872 => 6,616,864
        iOS: 6,562,352 => 6,605,152
      
      On top of ICU 59.1 from the upstream, the following changes were applied.
      See https://chromium.googlesource.com/chromium/deps/icu/+log/chromium/59staging
      
        - Fix C++ 11 string literal assignment issue (upstream bug: 13192)
        - Fix C4229 warning by MSVC
        - Apply utf32.patch and include unistr.h in fuzzer_util
        - Update ICU data files
        - Fix wpo.patch
        - Apply Google locale patch and locale1.patch
        - update readme
        - Apply breakiterator related patches
        - Apply and update wpo.patch
        - Drop unused patch, apply data.build.win.patch, update README.chromium
        - Add /utf-8 flag for Windows/Visual Studio
        - Update BUILD.gn for UChar, stubdata and apply data_sym.patch
        - use stubdata.cpp instead of stubdata.c in icu.gyp
        - Update icu.gyp* files for v8
        - Update BUILD.gn, apply data.build.patch and vscomp.patch
        - Add new files in ICU 59.1
        - Get a fresh copy of ICU 59.1 from the upstream
        - Update update.sh script
      
      TBR=drott@chromium.org, yangguo@chromium.org
      Bug:699469
      TEST: layout tests, all unittests, browser tests
      Change-Id: Ie1e77323aa0c7f872153680c4deca6471a771a5c
      Reviewed-on: https://chromium-review.googlesource.com/505173
      
      
      Reviewed-by: default avatarJungshik Shin <jshin@chromium.org>
      87232d8d
  2. Dec 09, 2016
    • Jungshik Shin's avatar
      Fix big endian support for MIPS · 73e24736
      Jungshik Shin authored
      Recent changes in detecting big endian targets don't work when cross
      compiling for a target with different endianness. This brakes MIPS.
      Added additional checks for big endian MIPS and MIPS64 targets to fix
      this.
      
      Patch by miran.karic@imgtec.com
      
      BUG=None
      R=bjaideep@ca.ibm.com, jshin@chromium.org
      
      Review URL: https://codereview.chromium.org/2518533002 .
      73e24736
  3. Nov 11, 2016
  4. Oct 28, 2016
    • Jungshik Shin's avatar
      ICU update to 58 part 2 · e0d9b90c
      Jungshik Shin authored
      Follw-up to https://chromium.googlesource.com/chromium/deps/icu/+/5feb9ad5
      (due to a rietveld issue, part 1 was manually pushed).
      
      Update ICU to 58.1 release from ICU 56.1 part2.
      
      Listed below a tiny subset of what's new in 58.1:
      
        1. Unicode 9.0 from Unicode 8.0
          - Updated character properties including Emoji data up to 4.0beta.
          - Updated grapheme/word/line breaking rules for Emoji sequences and others.
      
        2. CLDR 30.0.2 from CLDR 28
          - Numerous locale data updates/improvements
      
        3. Spoofing API changes
        4. Greek uppercasing support as a part of regular case-mapping API.
        5. Line breaking rule file format optimization. This change enables me
           to add CJ loose line breaking rules back (previously, it's dropped
           to save space) so that Blink can use it for CJ.
      
      See http://site.icu-project.org/download/58 for more details on ICU 58.1
      and http://site.icu-project.org/download/57 for more details on ICU 57.1
      
      For CLDR 30, see http://cldr.unicode.org/index/downloads/cldr-30 .
      
      The size impact:
         Non-Android: 10,127,200 => 10,128,624 (delta = 1,424 / 0.014%)
         Android: 6,563,152 => 6,571,936 (delta = 8,784 / 0.13%)
      
      Below are the list of changes made on top of the upstream ICU 58.1
      in reverse order. Most of these changes were made in 58staging branch
      to run trybots and cherry-picked back for this CL. See
         https://chromium.googlesource.com/chromium/deps/icu/+/log/chromium/58staging
         https://codereview.chromium.org/2447513002/ : cr+blink update cl with
             58staging branch head.
      
      * Fix a build on Win without std::string (v8)
      * Add ms932 alias to Shift_JIS
      
      * Apply Google-specific locale data patches
      
      * Fix a bug in scriptset
      
      * Update windows-1255 mapping
      
      * Disable C4333 warning by MSVC (harmless)
      
      * Apply and update utf32.patch and README.chromium
      
      * Update and apply vscomp.patch
        stringpiece patch removed. VS2015 seems to be fine with a redefinition.
      
      * Update pre-built ICU data files
         Update *local.mk with a new copyright line
      
      * Apply more patches
        The following patches were applied and updated: data_symb, vscomp, wpo
      
        The unnecessary part was dropped from vscomp
      
      * Update BUILD.gn and icu.gyp* files
      
      * Update android/brkitr.patch
      
      * Update and apply more patches
      
      * Update and apply cjdict.patch
         Apply data.build.patch
      
      * Delete obsolete patches: cmemory,regex
      
      * Update README.chromium and apply brkitr patches
      
        - Update README.chromium
        - Remove obsolete patches
        - Update linebrk.patch and apply it: add back line_loose_cj
      
      * Update wordbrk.patch and apply it
      
      * Update and apply khmer-dictbe.patch
      
      * Update data trimming
      
        - android/patch_locale.sh
        - scripts/trim_data.sh
           ExemplarCh* removed
           charac*Label removed
           relative/relativeTime removed for daysOfWeek and quarter
      
      * Update the following patches
      
        android/brkitr.patch
        patches/linebrk.patch
        patches/data.build.patch
      
      * Update cjdict.patch and linebrk.patch
      
      BUG=637001
      TEST=Layout tests, all unittests, browser tests, ui tests.
      R=jsbell@chromium.org, mark@chromium.org
      
      Review URL: https://codereview.chromium.org/2442923002 .
      e0d9b90c
  5. Sep 13, 2016
  6. Jul 22, 2016
  7. Jul 21, 2016
    • Miran Karic's avatar
      Add big endian support · e7d37b69
      Miran Karic authored
      Add a script that generates an assembly file from a .dat file. This is
      needed for generating big endian assembly file after using icupkg to
      convert little endian icudtl.dat to big endian icudtb.dat. Also the
      icu.gyp file is modified so big endian architectures use appropriate
      files.
      
      Patch by miran.karic@  ( https://codereview.chromium.org/1967523002/)
      with a couple of fixes:
      
      1. Two errors mentioned against PS#9 in the above CL.
      2. Support copying icu data file for Big Endian targets.
      
      Besides, icudtb.dat was added to common. icudtb.dat was created by
      running 'icupkg -tb icudt56l.dat icudt56b.dat' and renaming icudt56b.dat
      to icudtb.dat.
      
      BUG=v8:4828
      TEST='d8' is built correct with icu_use_data_file set to either 0 or 1.
      TEST=run `GYP_DEFINES="target_arch=mips" ./gypfiles/gyp_v8` and make sure
      that ninja files use 'b' data/assembly file for Big Endian.
      
      Review URL: https://codereview.chromium.org/2162393003 .
      
      Patch from Miran Karic <miran.karic@imgtec.com>.
      e7d37b69
  8. May 19, 2016
  9. Mar 25, 2016
  10. Mar 18, 2016
  11. Feb 02, 2016
  12. Jan 29, 2016
    • Jungshik Shin's avatar
      ICU 56 step 6:Check in the pre-built ICU data · d2c18300
      Jungshik Shin authored
      * Update the pre-built ICU data files for all platforms
      
        source/data/in/icudtl.dat for non-Android platforms
        {linux,mac}/icudt*.S for linux/mac
        android/icudtl.dat and android/icudt*.S for Android
        windows/icudt.dll for Windows
      
      * Update Android data trimming script
      
        1. Make sure that 'default' calendar is kept in locales where it's relevant
          : root, th, fa, ar_SA, etc.
        2. Add a minimal region data to work around a bug in ICU with pool.res
           handling
      
      * Update gn and gyp files
      * And add a TODO comment to update.sh to automate the build file update.
      * Add it_CH to the locale list.
      * Add sr_Latn to unit/reslocal.mk (required by sh) and
        line_normal_fi to brkitr/brklocal.mk (referred to in brkitr/fi.txt) in
        place of line_fi.
      
      * Update and add scripts for data building
      * Completely rewrite README.chromium
      * Check-in the prebuilt ICU data files/assembly sources for
        Linux,Mac,Windows,Chrome OS and Android.
      
      BUG=575007
      TEST=Blink layout tests, webkit unittests
      TEST=All bots can build successfully
      TEST=net_unittests --gtest_filter="*ilenameUtil*"
      TEST=net_unittests --gtest_filter="*IDN*" (pending bug 336973)
      TEST=base_unittests --gtest_filter="*Conv*"
      TEST=browser_tests --gtest_filter="*ncoding*"
      TEST=base_unittests --gtest_filter="*essage*"
      TEST=ui_base_unittests --gtest_filter="*ormat*"
      TEST=ui_base_unittests --gtest_filter="L10n*"
      R=mark@chromium.org
      
      Review URL: https://codereview.chromium.org/1639543006 .
      d2c18300
  13. Aug 18, 2015
  14. Jul 03, 2015
  15. Jul 01, 2015
  16. Jun 30, 2015
  17. Jun 01, 2015
    • Jungshik Shin's avatar
      Remove unnecessary clang flags for ICU · dbed54da
      Jungshik Shin authored
      1. -Wno-return-type-c-linkage
       - became unnecessary with ICU bug 9504 fixed 3 years ago.
       - remove from both BUILD.gn and icu.gyp
      
      2. -Wnoheader-hygine
       - became unnecessary a long time ago.
       - remove from BUILD.gn (icu.gyp does not have it)
      
      3. -Wstring-conversion and -Wheader-hygine
       - add to BUILD.gn by NOT excluding clang:extra_warnings
      
      BUG=114660
      TEST=Build icuuc/icui18n targets (both gyp and GN) after regenerating build files.
      R=brettw@chromium.org, thakis@chromium.org
      
      Review URL: https://codereview.chromium.org/1158933005
      dbed54da
  18. Mar 19, 2015
    • Jungshik Shin (jungshik at google)'s avatar
      Update CJK converters and their generating scripts · dafa8443
      Jungshik Shin (jungshik at google) authored
      1. Update ucmlocal.mk and convertrs.txt to refer to euc-kr-html.ucm
      instead of windows-949.ucm
      
      2. Tighten up the valid code range for the following converters:
      
         EUC-KR, Shift_JIS, Big5
      
      This is to add back an ASCII range byte to the stream per
      the encoding spec when they're either illegal as a 'trail byte' or
      there's no assigned code point for a "lead + trail" sequence.
      For instance, with this change, '0xF3 0x41' in EUC-KR is converted to
      'U+FFFD U+0041' instead of 'U+FFFD'.
      
      This change requires adding 2 ~ 8 new states to the conversion
      table of each converter mentioned above leading to 6.5kB net increase
      in the final data size.
      
      3. Tighten the trail byte range for 2-byte sequences starting with 0x8E
      from [A1,E2] to [A1,DF] in EUC-JP and update the corresponding generating
      script.
      
      4. Change the substitution characters for EUC-JP and Shift_JIS to
      match other converters. i.e. make them produce U+FFFD when encountering
      an invalid input. Before this chaange, they emitted U+001A.
      
      5. Enable 'U_CHARSET_IS_UTF8' configuration flag.
      Chromium/Blink does not rely on ICU for the code conversion between
      the 'system native encoding' (if it's one of legacy encodings)
      and Unicode. With this configuration, we can cut down the code size
      a bit.
      
      6. Update the icudtl.dat (all platforms) and assembly files (mac,linux)
         and the icudata dll (windows)
      
      See https://codereview.chromium.org/1026453002 for a new blink test
      added ( fast/encoding/char-decoding-invalid-trail.html )
      
      BUG=450312,430823
      TEST=Blink: fast/encoding/char-decoding-{truncated,invalid-trail}.html
      TEST=base_unittests --gtest_filter=*Conv*, browser_tests --gtest_filter=*ncoding*
      R=jsbell@chromium.org, mark@chromium.org
      
      Review URL: https://codereview.chromium.org/984233002
      dafa8443
  19. Feb 17, 2015
  20. Feb 14, 2015
    • Jungshik Shin (jungshik at google)'s avatar
      Move stubdata.c from icudata to icuuc · d158fece
      Jungshik Shin (jungshik at google) authored
      This is to fix a linker error when linking icuuc.dll;  ICU_DATA_ENTRY (icudt54_dat) symbol is not found on Windows clean build from the scratch (component=shared_library).
      
      Move stubdata.c to icuuc target from icudata target. Also, make
      U_DATA_API (used for U_ICU_DATA_ENTRYPOINT in common/udata.cpp) to be
      U_EXPORT instead of U_IMPORT when icu_use_data_file_flag = 1 or
      on Windows. On Windows, using the icudt.dll  (i.e. icu_use_data_file_flag=0) also requires this change.
      
      BUG=428145
      TEST=All trybots can build a target that requires ICU.
      R=mark@chromium.org, scottmg@chromium.org
      
      Review URL: https://codereview.chromium.org/926113004
      d158fece
  21. Jan 08, 2015
  22. Oct 30, 2014
  23. Sep 25, 2014
    • jshin@chromium.org's avatar
      Turn on UCONFIG_NO_NON_HTML5_CONVERTER · 52e8245c
      jshin@chromium.org authored
      UCONFIG_NO_NON_HTML5_CONVERTER was added earlier to our copy of ICU, but
      it was never set to 1.  It's my oversight.
      
      1. Turns UCON..CONVERTER on in icu.gyp to drop all the encodings not
         required by the Encoding spec. Dropped encodings include
         UTF-7, BOCU, SCSU, CESU, ISCII, ISO-2022-{KR, CN*}, HZ-GB, ISO-2022-JP's
         other than the original.
      
      2. A lot more sections of the ICU converter code are excluded when
         it's set to 1 including the code for LMB (Lotus Multibyte) encodings and
         X11 compound text encoding (icu common).
      
      3. The character encoding detections for encodings excluded are also disabled.
         (icu i18n)
      
      4. ISO-2022-{KR, CN*} and HZ-GB can be dropped now because Blink treats them
         as replacement encoding. The corresponding alias entries from convertrs.txt
         are also removed.
      
      5. ibm-874 was removed. We used to need it before Blink started, but not any
         more. We only need windows-874.
      
      6. A mistaken in convertrs.txt was corrected : Big5-HKSCS was pointing to
         an old mapping table.
      
      7. Per ICU upstream's suggestion, use '-html' suffix instead of '-html5'
      for the encoding tables derived from the WHATWG's encoding spec (ibm866,
      shift_jis and euc-jp).
      
      The static 64-bit release build of Chrome on Linux went down from
      141,596,616 to 141,491,968 bytes (~ 100 kB reduction). Besides, the icu data
      size got smaller by ~ 19 kB ( 10,490,576 to 10,471,008 bytes).
      
      See http://bugs.icu-project.org/trac/ticket/11296 for an upstream bug
      I've filed on the issue.
      
      
      BUG=76328
      TEST=browser_tests --gtest_filter="*ncoding*"
      TEST=net_unittest --gtest_filter="*ilenameUtil*"
      TEST=base_unittests --gtest_filter="*Conv*"
      TEST=Blink: fast/encoding/*
      TEST=With shared library build, the following has no match.
        nm libicuuc.so | egrep  -i '(bocu|scsu|utf7|2022kr|2022cn|iscii)'
        nm libicui18n.so | egrep  -i '(2022kr|2022cn|ibm42)'
      TEST=With static library build, the following has no match.
        nm chrome | egrep -i '(bocu|scsu|utf7|2022kr|2022cn|iscii|ibm42)'
      
      R=jsbell@chromium.org, mark@chromium.org
      
      Review URL: https://codereview.chromium.org/587833004
      
      git-svn-id: http://src.chromium.org/svn/trunk/deps/third_party/icu52@292131 4ff67af0-8c30-449e-8e8b-ad334ec8d88c
      52e8245c
  24. Sep 03, 2014
    • torne@chromium.org's avatar
      Remove invalid link_settings from target condition. · 89831135
      torne@chromium.org authored
      ICU fails to gyp when run with GYP_DEFINES="android_webview_build=1
      use_system_icu=0 use_system_stlport=1" which is a combination of
      settings we're trying to bring up temporarily as we migrate away from
      system libraries. It fails because it's not permitted to specify
      link_settings in a target_condition as the processing is too late.
      
      Remove the invalid link_settings, since we can't do this outside
      target_conditions as it's not valid to use -lgabi++ on the host build of
      ICU. The link dependency on gabi++ will have to be satisfied manually in
      the main libwebviewchromium target instead for this configuration.
      
      BUG=409851
      R=mkosiba@chromium.org
      TBR=jshin@chromium.org
      
      Review URL: https://codereview.chromium.org/527193003
      
      git-svn-id: http://src.chromium.org/svn/trunk/deps/third_party/icu52@291781 4ff67af0-8c30-449e-8e8b-ad334ec8d88c
      89831135
  25. Aug 28, 2014
  26. Aug 21, 2014
  27. Aug 01, 2014
  28. Jul 24, 2014
  29. Jun 18, 2014
  30. Jun 10, 2014
  31. Apr 30, 2014
  32. Apr 18, 2014
    • jshin@chromium.org's avatar
      Trim ICU data to reduce the download size/memory usage · 4e493261
      jshin@chromium.org authored
      Add a shell script to trim the ICU data further : trim_data.sh along with
      locale list files.  The script does the following:
      
      1. Remove the display names of languages NOT listed in Chrome's Accept-Language
         list. (800kB)
      2. Minimize the locale data for locales listed in the A-L list that are
         not a UI locale in Chrome. For those locales, exemplar characters,
         the display name in the native language and layout direction are included.
         (640kB)
      3. Filter the region data to drop numeric region display names other than 419
         (Latin-America). (50kB)
      4. Filter the currency data (display name and plurals) for historic currencies.
         (200kB)
      
      This CL also checks in icudtl.dat (source/data/in) and
      icudt_dat.S (mac and linux). Note that I dropped '52' (the version number)
      in the assembly source file name and icu.gyp was adjusted accordingly.
      
      With all these changes, icudtl.dat is ~ 800kB larger than that in ICU 4.6.
      The 7z compression (as used by the installer) makes the size difference
      go down to ~ 130kB.
      
      BUG=132145
      TEST=The icudtl.dat (uncompressed) is about 10.7MB instead of 12.4MB without this CL.
      R=mark@chromium.org
      
      Review URL: https://codereview.chromium.org/239543018
      
      git-svn-id: http://src.chromium.org/svn/trunk/deps/third_party/icu52@264811 4ff67af0-8c30-449e-8e8b-ad334ec8d88c
      4e493261
  33. Apr 01, 2014
Loading