ProductType Sync

The module used for importing/syncing ProductTypes into a commercetools project. It also provides utilities for generating update actions based on the comparison of a ProductType against a ProductTypeDraft.

• Usage
• Prerequisites
  • ProjectApiRoot
  • Required Fields
  • Reference Resolution
  • Syncing from a commercetools project
  • Syncing from an external resource
  • SyncOptions
  • errorCallback
  • warningCallback
  • beforeUpdateCallback
  • beforeCreateCallback
  • batchSize
  • cacheSize
• Running the sync
  • Important to Note
  • More examples of how to use the sync
• Build all update actions
• Build particular update action(s)
• Caveats
• Migration Guide
• Client configuration and creation
• Signature of ProductTypeSyncOptions
• Build ProductTypeDraft (syncing from external project)
• Query for ProductTypes (syncing from CTP project)
• JVM-SDK-V2 migration guide

Usage

Prerequisites

ProjectApiRoot

Use the ClientConfigurationUtils which apply the best practices for ProjectApiRoot creation. To create the required ClientCredentials for client creation, please utilize the ClientCredentialsBuilder provided in the java-sdk-v2 Client OAUTH2 package. If you have custom requirements for the client creation, have a look into the Important Usage Tips.

final ClientCredentials clientCredentials =
        new ClientCredentialsBuilder()
        .withClientId("client-id")
        .withClientSecret("client-secret")
        .withScopes("scopes")
        .build();
final ProjectApiRoot apiRoot = ClientConfigurationUtils.createClient("project-key", clientCredentials, "auth-url", "api-url");

Required Fields

The following fields are required to be set in, otherwise, they won't be matched by sync:

Draft	Required Fields	Note
ProductTypeDraft	key	Also, the product types in the target project are expected to have the key fields set.

Reference Resolution

In commercetools, a reference can be created by providing the key instead of the ID with the type ResourceIdentifier. When the reference key is provided with a ResourceIdentifier, the sync will resolve the resource with the given key and use the ID of the found resource to create or update a reference. Therefore, in order to resolve the actual ids of those references in the sync process, ResourceIdentifiers with their keys have to be supplied.

Reference Field	Type
attributes	Only the attributes with type AttributeNestedType and AttributeSetType with elementType as AttributeNestedType requires key on the id field of the ProductTypeReference.

Note that a reference without the key field will be considered as an existing resource on the target commercetools project and the library will issue an update/create an API request without reference resolution.

Syncing from a commercetools project

When syncing from a source commercetools project, you can use toProductTypeDrafts method that transforms(resolves by querying and caching key-id pairs) and maps from a ProductType to ProductTypeDraft. It can be configured to use a cache that will speed up the reference resolution performed during the sync, for example:

// Build ByProjectKeyProductTypesGet for fetching product-types from a source CTP project without any references expanded for the sync:
final ByProjectKeyProductTypesGet byProjectKeyProductTypesGet = client.productTypes().get();

// Query all product-types (NOTE this is just for example, please adjust your logic)
final List<ProductTypes> productTypes = QueryUtils.queryAll(byProjectKeyProductTypesGet,
            (productTypes) -> productTypes)
            .thenApply(lists -> lists.stream().flatMap(List::stream).collect(Collectors.toList()))
            .toCompletableFuture()
            .join();

In order to transform and map the ProductType to ProductTypeDraft, Utils method toProductTypeDrafts requires projectApiRoot, implementation of ReferenceIdToKeyCache and a list of productTypes as parameters. For cache implementation, you have two options: you can either use your own cache implementation or utilize the class CaffeineReferenceIdToKeyCacheImpl provided in the library. This class implements the cache using caffeine library with an LRU (Least Recently Used) based cache eviction strategy.Example as shown below:

//Implement the cache using library class.
final ReferenceIdToKeyCache referenceIdToKeyCache = new CaffeineReferenceIdToKeyCacheImpl();

//For every reference fetch its key using id, cache it and map from ProductType to ProductTypeDraft. With help of the cache same reference keys can be reused.
CompletableFuture<List<ProductTypeDraft>> productTypeDrafts = ProductTransformUtils.toProductTypeDrafts(client, referenceIdToKeyCache, productTypes);

Syncing from an external resource

• Attributes with type AttributeNestedType do not support the ResourceIdentifier yet, for those references you have to provide the key value on the id field of the reference. This means that calling getId() on the reference should return its key.

final AttributeDefinitionDraft nestedTypeAttr = AttributeDefinitionDraftBuilder.of()
            .type(AttributeNestedTypeBuilder.of()
                    .typeReference(ProductTypeReferenceBuilder.of().id("product-type-key").build()) // note that key is provided in the id field of reference
                    .build())
            .name("attrNestedType")
            .label(LocalizedString.ofEnglish("attrNestedType"))
            .isRequired(true)
            .build();

final AttributeDefinitionDraft setOfNestedTypeAttr = AttributeDefinitionDraftBuilder.of()
            .type(AttributeSetTypeBuilder.of()
                    .elementType(AttributeNestedTypeBuilder.of()
                              .typeReference(ProductTypeReferenceBuilder.of()
                              .id("product-type-key").build())
                    .build())
                  .build())
            .name("attrNestedType")
            .label(LocalizedString.ofEnglish("attrNestedType"))
            .isRequired(true)
            .build();

final ProductTypeDraft productTypeDraft =
    ProductTypeDraftBuilder.of()
            .name("foo")
            .description("description")
            .key("key")
            .attributes(nestedTypeAttr, setOfNestedTypeAttr)
            .build();

SyncOptions

After the projectApiRoot is setup, a ProductTypeSyncOptions should be built as follows:

// instantiating a ProductTypeSyncOptions
final ProductTypeSyncOptions productTypeSyncOptions = ProductTypeSyncOptionsBuilder.of(projectApiRoot).build();

SyncOptions is an object which provides a place for users to add certain configurations to customize the sync process. Available configurations:

errorCallback

A callback that is called whenever an error event occurs during the sync process. Each resource executes its own error-callback. When the sync process of a particular resource runs successfully, it is not triggered. It contains the following context about the error-event:

• sync exception
• product type draft from the source
• product type of the target project (only provided if an existing product type could be found)
• the update-actions, which failed (only provided if an existing product type could be found)

 final Logger logger = LoggerFactory.getLogger(ProductTypeSync.class);
 final ProductTypeSyncOptions productTypeSyncOptions = ProductTypeSyncOptionsBuilder
         .of(projectApiRoot)
         .errorCallback((syncException, draft, productType, updateActions) -> 
            logger.error(new SyncException("My customized message"), syncException)).build();

warningCallback

A callback is called whenever a warning event occurs during the sync process. Each resource executes its own warning-callback. When the sync process of a particular resource runs successfully, it is not triggered. It contains the following context about the warning message:

• sync exception
• product type draft from the source
• product type of the target project (only provided if an existing product type could be found)

 final Logger logger = LoggerFactory.getLogger(ProductTypeSync.class);
 final ProductTypeSyncOptions productTypeSyncOptions = ProductTypeSyncOptionsBuilder
         .of(projectApiRoot)
         .warningCallback((syncException, draft, productType) -> 
            logger.warn(new SyncException("My customized message"), syncException)).build();

beforeUpdateCallback

During the sync process, if a target product type and a product type draft are matched, this callback can be used to intercept the update request just before it is sent to the commercetools platform. This allows the user to modify update actions array with custom actions or discard unwanted actions. The callback provides the following information :

• product type draft from the source
• product type from the target project
• update actions that were calculated after comparing both

final TriFunction<
        List<ProductTypeUpdateAction>, ProductTypeDraft, ProductType, List<ProductTypeUpdateAction>> 
            beforeUpdateProductTypeCallback =
            (updateActions, newProductTypeDraft, oldProductType) ->  updateActions.stream()
                    .filter(updateAction -> !(updateAction instanceof ProductTypeRemoveAttributeDefinitionAction))
                    .collect(Collectors.toList());

final ProductTypeSyncOptions productTypeSyncOptions = 
        ProductTypeSyncOptionsBuilder.of(projectApiRoot).beforeUpdateCallback(beforeUpdateProductTypeCallback).build();

beforeCreateCallback

During the sync process, if a product type draft should be created, this callback can be used to intercept the create request just before it is sent to the commercetools platform. It contains the following information :

• product type draft that should be created

Please refer to example in product sync document.

batchSize

A number that could be used to set the batch size with which product types are fetched and processed, as product types are obtained from the target project on the commercetools platform in batches for better performance. The algorithm accumulates up to batchSize resources from the input list, then fetches the corresponding product types from the target project on the commecetools platform in a single request. Playing with this option can slightly improve or reduce processing speed. If it is not set, the default batch size is 50 for product type sync.

final ProductTypeSyncOptions productTypeSyncOptions = 
         ProductTypeSyncOptionsBuilder.of(projectApiRoot).batchSize(30).build();

cacheSize

In the service classes of the commercetools-sync-java library, we have implemented an in-memory LRU cache to store a map used for the reference resolution of the library. The cache reduces the reference resolution based calls to the commercetools API as the required fields of a resource will be fetched only one time. These cached fields then might be used by another resource referencing the already resolved resource instead of fetching from commercetools API. It turns out, having the in-memory LRU cache will improve the overall performance of the sync library and commercetools API. which will improve the overall performance of the sync and commercetools API.

Playing with this option can change the memory usage of the library. If it is not set, the default cache size is 10.000 for product type sync.

final ProductTypeSyncOptions productTypeSyncOptions = 
         ProductTypeSyncOptionsBuilder.of(projectApiRoot).cacheSize(5000).build();

Running the sync

After all the aforementioned points in the previous section have been fulfilled, to run the sync:

// instantiating a product type sync
final ProductTypeSync productTypeSync = new ProductTypeSync(productTypeSyncOptions);

// execute the sync on your list of product types
CompletionStage<ProductTypeSyncStatistics> syncStatisticsStage = productTypeSync.sync(productTypeDrafts);

The result of completing the syncStatisticsStage in the previous code snippet contains a ProductTypeSyncStatistics which contains all the stats of the sync process; which includes a report message, the total number of updated, created, failed, processed product types and the processing time of the last sync batch in different time units and in a human-readable format.

final ProductTypeSyncStatistics stats = syncStatisticsStage.toCompletebleFuture().join();
stats.getReportMessage();
/*"Summary: 2000 products types were processed in total (1000 created, 995 updated, 5 failed to sync)."*/

Note The statistics object contains the processing time of the last batch only. This is due to two reasons:

1. The sync processing time should not take into account the time between supplying batches to the sync.
2. It is not known by the sync which batch is going to be the last one supplied.

Important to Note

1. If two matching attributeDefinitions (old and new) on the matching productTypes (old and new) have a different AttributeType, the sync will remove the existing attributeDefinition and then add a new attributeDefinition with the new AttributeType.

2. The attributeDefinition for which the AttributeType is not defined (null) will not be synced.

More examples of how to use the sync

1. Sync from another CTP project as a source.
2. Sync from an external source.

Make sure to read the Important Usage Tips for optimal performance.

Build all update actions

A utility method provided by the library to compare a ProductType with a new ProductTypeDraft and results in a list of product type update actions.

List<ProductTypeUpdateAction> updateActions = ProductTypeSyncUtils.buildActions(productType, productTypeDraft, productTypeSyncOptions);

Build particular update action(s)

Utility methods provided by the library to compare the specific fields of a ProductType and a new ProductTypeDraft, and in turn, build the update action. One example is the buildChangeNameUpdateAction which compares names:

Optional<ProductTypeUpdateAction> updateAction = ProductTypeUpdateActionUtils.buildChangeNameAction(oldProductType, productTypeDraft);

More examples of those utils for different fields can be found here.

Caveats

1. The order of attribute definitions in the synced product types is not guaranteed.
2. Changing the attribute definition type is not supported. Instead, remove and re-add it with a new type manually, either over API or merchant center. For more information please check this ADR.

Migration Guide

The producttype-sync uses the JVM-SDK-V2, therefore ensure you Install JVM SDK module commercetools-sdk-java-api with any HTTP client module. The default one is commercetools-http-client.

 <!-- Sample maven pom.xml -->
 <properties>
     <commercetools.version>LATEST</commercetools.version>
 </properties>

 <dependencies>
     <dependency>
       <groupId>com.commercetools.sdk</groupId>
       <artifactId>commercetools-http-client</artifactId>
       <version>${commercetools.version}</version>
     </dependency>
     <dependency>
       <groupId>com.commercetools.sdk</groupId>
       <artifactId>commercetools-sdk-java-api</artifactId>
       <version>${commercetools.version}</version>
     </dependency>
 </dependencies>

Client configuration and creation

For client creation use ClientConfigurationUtils which apply the best practices for ProjectApiRoot creation. If you have custom requirements for the client creation make sure to replace SphereClientFactory with ApiRootBuilder as described in this Migration Document.

Signature of ProductTypeSyncOptions

As models and update actions have changed in the JVM-SDK-V2 the signature of SyncOptions is different. It's constructor now takes a ProjectApiRoot as first argument. The callback functions are signed with ProductTypeDraft, ProductType and ProductTypeUpdateAction from package com.commercetools.api.models.product_type.*

Note: Type UpdateAction<ProductType> has changed to ProductTypeUpdateAction. Make sure you create and supply a specific ProductTypeUpdateAction in beforeUpdateCallback. For that you can use the library-utilities or use a JVM-SDK builder (see also):

// Example: Create a producttype update action to change name taking the 'newName' of the productTypeDraft
    final Function<LocalizedString, ProductTypeUpdateAction> createBeforeUpdateAction =
        (newName) -> ProductTypeChangeNameAction.builder().name(newName).build();

// Add the change name action to the list of update actions before update is executed
    final TriFunction<
            List<ProductTypeUpdateAction>, ProductTypeDraft, ProductType, List<ProductTypeUpdateAction>>
        beforeUpdateProductTypeCallback =
            (updateActions, newProductTypeDraft, oldProductType) -> {
              final ProductTypeUpdateAction beforeUpdateAction =
                  createBeforeUpdateAction.apply(newProductTypeDraft.getName());
              updateActions.add(beforeUpdateAction);
              return updateActions;
            };

Build ProductTypeDraft (syncing from external project)

The producttype-sync expects a list of ProductTypeDrafts to process. If you use java-sync-library to sync your producttypes from any external system into a commercetools platform project you have to convert your data into CTP compatible ProductTypeDraft type. This was done in previous version using DraftBuilders. The V2 SDK do not have inheritance for DraftBuilder classes but the differences are minor and you can replace it easily. Here's an example:

// SDK v1: ProductTypeDraftBuilder.of  takes parameters 'key', 'name', 'description' and list of 'attributes'
final ProductTypeDraft productTypeDraft =
              ProductTypeDraftBuilder
                      .of("key", "name", "description", emptyList())
                      .build();

// SDK v2: ProductTypeDraftBuilder without draftTemplate
final ProductTypeDraft productTypeDraft =
              ProductTypeDraftBuilder.of()
                      .description("product-type-description")
                      .key("product-type-key")
                      .name("product-type-name")
                      .build();

For more information, see the Guide to replace DraftBuilders.

Query for ProductTypes (syncing from CTP project)

If you sync producttypes between different commercetools projects you probably use ProductTypeTransformUtils#toProductTypeDrafts to transform ProductType into ProductTypeDraft which can be used by the producttype-sync. However, if you need to query ProductTypes from a commercetools project instead of passing ProductTypeQuerys to a sphereClient, create (and execute) requests directly from the apiRoot. Here's an example:

// SDK v1: ProductTypeQuery to fetch all producttypes
final ProductTypeQuery query = ProductTypeQuery.of();

final PagedQueryResult<ProductType> pagedQueryResult = sphereClient.executeBlocking(query);

// SDK v2: Create and execute query to fetch all producttypes in one line
final ProductTypePagedQueryResponse result = apiRoot.productTypes().get().executeBlocking().getBody();

Read more about querying resources.

JVM-SDK-V2 migration guide

On any other needs to migrate your project using jvm-sdk-v2 please refer to its Migration Guide.