Add the country translation engines (ip-intelligence.translation)#136
Open
Jamesr51d wants to merge 1 commit into
Open
Add the country translation engines (ip-intelligence.translation)#136Jamesr51d wants to merge 1 commit into
Jamesr51d wants to merge 1 commit into
Conversation
Add a new ip-intelligence.translation module with the two country translation engines, ported from the .NET FiftyOne.IpIntelligence.Translation package. Engine 1 (CountryCodeTranslationEngine) reads the weighted ISO country codes from the IP Intelligence engine and translates them to English country names using countrycodes.en_GB.yml. Its source element data key is configurable and defaults to "ip". Engine 2 (CountriesTranslationEngine) translates those English names to the browser language using the countries.<locale>.yml files, then builds complete ordered lists: the weighted countries first (most probable by weight, ties broken by translated name), followed by every remaining country alphabetically. The code and name "All" lists are index aligned so a demo can render a country dropdown with the most probable country first. The translation YAML files are wired in at build time from the IP Intelligence data submodule rather than shipped as new data.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add the country translation engines (ip-intelligence.translation)
Background
On-premise IP Intelligence returns
CountryCodesGeographicalandCountryCodesPopulationas weighted lists of ISO country codes. The demos needlocalized, complete, ordered country name and code lists so they can render a
country dropdown with the most probable country at the top. Cloud already
returns these properties in final form, so this work is on the on-premise path
only and the cloud engine is untouched. This is the Java port of the .NET
FiftyOne.IpIntelligence.Translationpackage, building on the newpipeline.translationmodule.Objectives
country names.
complete, index-aligned, ordered "All" lists (most probable countries first
by weight, then every remaining country alphabetically by translated name).
Accept-Language(or aquery.translationoverride), defaultingto English when the language is English or unknown.
Key Decisions
ip-intelligence.translation, depending on the newpipeline.translation, plusip-intelligence.sharedfor theIPIntelligenceDatainterface.and cloud engines return
"ip", so that is the default source key forEngine 1. Because the key has changed between engine versions, Engine 1's
source key is configurable
(
CountryCodeTranslationEngineBuilder.setSourceElementDataKey, default"ip"). Engine 2 reads the weighted codes byIPIntelligenceDatatype, so itis key-agnostic.
build time from the IP Intelligence data submodule
(
ip-intelligence-cxx/ip-intelligence-data/Translations) into the jar,mirroring the .NET project which links the same files.
Resourcesloads the files from the classpath by their known names, soloading is reliable from a directory or a packaged jar.
Changes
ip-intelligence.translationmodule:Constants,Resources.data:ICountryCodeTranslationData/CountryCodeTranslationData,ICountriesTranslationData/CountriesTranslationData.flowelements:CountryCodeTranslationEngine(+ builder, configurablesource key),
CountriesTranslationEngine(+ builder) with the orderingalgorithm and the locale-aware
Collatorsort, recording the sortingculture in
SortingCultureUsed.pom.xml: register the module in<modules>; copy the translation YAML fromthe submodule at build time.
Testing
mvn -pl ip-intelligence.translation test(JDK 17, source/target 1.8). 22tests, all passing, PMD and
-Werrorclean.TranslationTests(21): the 12 tests ported from the .NET suite plus 9 gaptests (evidence-key precedence including
query.translationoverriding theheader, weights preserved through translation, equal-weight tie break by
name, unknown locale falling back to English, a missing single name staying
English, full index alignment across the alphabetical tail,
SortingCultureUsed, and population names differing from geographical).These use a real mini pipeline (a stub IP engine, then the two engines).
CountryTranslationIntegrationTest(1): drives the real on-premise enginewith the 6 GB enterprise data file, then the two engines. With
8.8.8.8andfr_FRthe dropdown leads with the most probable country (Pakistan in thisdata) and is followed by all 250 countries alphabetically in French, with the
code and name lists index-aligned. Skipped automatically when no data file is
present. Run it with
-DTestDataFile=<path to a .ipi>(or set51DEGREES_IPI_PATH).JDK note: build this module with JDK 17. The repo's PMD-on-compile check uses
an ASM that cannot parse JDK 25 class files.
Notes
<modules>.The on-premise engine and the cloud path are unchanged.
pipeline.translation,a new module in
pipeline-java.ip-intelligence-javaconsumes pipeline atthe published
${pipeline.version}(currently 4.5.10), while the localpipeline-javasource is 4.5.7-SNAPSHOT. So:pipeline.translationat${pipeline.version},consistent with the other pipeline dependencies.
pipeline-javapublishespipeline.translationin the pipeline line this repo targets. Merge orrelease the
pipeline-javachange first.pipeline-javaworktree and installed into the local Maven repo, then also installed under
version 4.5.10 (a binary-compatible bridge, verified with javap) so this
repo resolves it at
${pipeline.version}.example with an ordering test live in
ip-intelligence-java-examplesandconsume this module.