Environment configuration before iServer uses Hadoop Yarn cluster

Feedback

If you already have an available Hadoop Yarn cluster (including a Hadoop Yarn cluster with Kerberos authentication enabled and a Hadoop Yarn cluster without Kerberos authentication enabled), this chapter will walk you through the environment preparation before using the Hadoop Yarn cluster in the distributed analysis service.

Configurations before using a Hadoop Yarn cluster in the distributed analysis service

If you're using a Hadoop Yarn cluster with Kerberos authentication enabled, you need to perform all the steps below; if you're using a a Hadoop Yarn cluster without Kerberos authentication enabled, you only need to configure steps 3 and 4.

  1. Install the Kerberos client on the machine where the iServer service is located. The time difference between the client and the server cannot exceed 5 minutes:
  2. Configuring the Kerberos client
    1. If the iServer service is on Windows, you need to modify the C:\ProgramData\MIT\Kerberos5\krb5.ini file. The Kerberos information set must be the same as the Kerberos service configuration (krb5.conf). The specific contents are as follows:

[libdefaults]

            default_realm = SUPERMAP.COM

[realms]

            SUPERMAP.COM = {

               kdc = 192.168.112.162

               admin_server = 192.168.112.162

}

    1. If the iServer service is on Linux, you need to modify etc/krb5.conf. The content is the same as the Kerberos service configuration (krb5.conf).
  1. Add the following parameters to the spark-default.conf in the iServer's built-in spark (iServer installation package\support\spark):

spark.yarn.am.memory    4g

This parameter represents the memory size of the application master. If not set, the default is 512M. It is recommended to modigy it to 4G to ensure that the iServer distributed analysis service can be used normally. Users can allocate according to the memory of their own machines.

If you are using the Windows version of iServer, you need to add the following configuration:

spark.hadoop.yarn.resourcemanager.hostname   hostname

Among which, the hostname should be set to the hostname of the primary node of the yarn cluster. For more configurations, see:  http://spark.apache.org/docs/latest/running-on-yarn.html.

  1. Copy the {HADOOP_HOME}/etc/hadoop directory from the Hadoop Yarn cluster master to any location on the machine where the iServer service is located.
  2. Run the following commands on the root user of the master node of the Hadoop Yarn cluster:

kadmin.local

#Create a user

addprinc -randkey iserver@SUPERMAP.COM

  1. #Generate a key file (generated to the current path)

xst  -k yarn.keytab  iserver@SUPERMAP.COM

Copy the generated key file to any location on the same computer as the iServer service.