Re: How to use SKIP_SCHEMA_INIT=TRUE from command line

2023-12-22 Thread Sanjay Gupta 
Thanks, it solves issue. Much appreciated.


Thanks
Sanjay Gupta

From: Akshat m 
Sent: Friday, December 22, 2023 5:55:38 AM
To: user@hive.apache.org 
Subject: Re: How to use SKIP_SCHEMA_INIT=TRUE from command line

Hi Sanjay,

Instead of using  --env SKIP_SCHEMA_INIT=TRUE,

Please use --env IS_RESUME="true" while running,

docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore
--add-host=host.docker.internal:host-gateway \
 --env IS_RESUME="true" \
 --env DB_DRIVER=mysql \
 --env 
SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
 
-Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
-Djavax.jdo.option.ConnectionUserName=hive
-Djavax.jdo.option.ConnectionPassword=" \
 --mount source=warehouse,target=/opt/hive/data/warehouse \
 --name metastore-standalone apache/hive:${HIVE_VERSION} /bin/bash

This should skip the schema initOrUpgrade process: 
https://github.com/apache/hive/blob/5022b85b5f50615f85da07bce42aebd414deb9b0/packaging/src/docker/entrypoint.sh#L24
from the 2nd time you run the container.

Regards,
Akshat


On Fri, Dec 22, 2023 at 11:53 AM Sanjay Gupta 
mailto:sanja...@gmail.com>> wrote:
Hi All,

If my metastore schema already exists with correct version so what I
need to do so it doesn't do init or upgrade when starting metastore
container

I have tried following command line


On MAC environment variables
export HIVE_VERSION=3.1.3
and even
SKIP_SCHEMA_INIT=TRUE

docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore
--add-host=host.docker.internal:host-gateway \
 --env SKIP_SCHEMA_INIT=TRUE \
 --env DB_DRIVER=mysql \
 --env 
SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
 
-Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
-Djavax.jdo.option.ConnectionUserName=hive
-Djavax.jdo.option.ConnectionPassword=" \
 --mount source=warehouse,target=/opt/hive/data/warehouse \
 --name metastore-standalone apache/hive:${HIVE_VERSION} /bin/bash



---

Docker Logs , it still tries to initSchema

docker logs 1c
+ : mysql
+ SKIP_SCHEMA_INIT=false
+ export HIVE_CONF_DIR=/opt/hive/conf
+ HIVE_CONF_DIR=/opt/hive/conf
+ '[' -d '' ']'
+ export 'HADOOP_CLIENT_OPTS= -Xmx1G
-Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
-Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
-Djavax.jdo.option.ConnectionUserName=hive
-Djavax.jdo.option.ConnectionPassword=hive'
+ HADOOP_CLIENT_OPTS=' -Xmx1G
-Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
-Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
-Djavax.jdo.option.ConnectionUserName=hive
-Djavax.jdo.option.ConnectionPassword=hive'
+ [[ false == \f\a\l\s\e ]]
+ initialize_hive
+ COMMAND=-initOrUpgradeSchema
++ cut -d . -f1
++ echo 3.1.3
+ '[' 3 -lt 4 ']'
+ COMMAND=-initSchema
+ /opt/hive/bin/schematool -dbType mysql -initSchema
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/opt/hive/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:
jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
Metastore Connection Driver : com.mysql.cj.jdbc.Driver
Metastore connection User: hive
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.mysql.sql


Error: Table 'ctlgs' already exists (state=42S01,code=1050)
org.apache.hadoop.hive.metastore.HiveMetaException: Schema
initialization FAILED! Metastore state would be inconsistent !!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
Use --verbose for detailed stacktrace.
*** schemaTool failed ***
[WARN] Failed to create directory: /home/hive/.beeline
No such file or directory
+ '[' 1 -eq 0 ']'
+ echo 'Schema initialization failed!'
Schema initialization failed!
+ exit 1
-



Docker entrypoint.sh have following code

SKIP_SCHEMA_INIT="${IS_RESUME:-false}"

function initialize_hive {
  COMMAND="-initOrUpgradeSchema"
  if [ "$(echo "$HIVE_VER" | cut -d '.' -f1)" -lt "4" ]; then
 COMMAND="-${SCHEMA_COMMAND:-initSchema}"
  fi
  $HIVE_HOME/bin/schematool -dbType $DB_DRIVER $COMMAND
  if [ $? -eq 0 ]; then
echo "Initialized schema successfully.."
  else
echo "Schema initialization failed!"
exit 1
  fi
}

export HIVE_CON

Re: How to use SKIP_SCHEMA_INIT=TRUE from command line

2023-12-22 Thread Akshat m 
Hi Sanjay,

Instead of using  --env SKIP_SCHEMA_INIT=TRUE,

Please use --env IS_RESUME="true" while running,

docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore
--add-host=host.docker.internal:host-gateway \
 --env IS_RESUME="true" \
 --env DB_DRIVER=mysql \
 --env
SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
 
-Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
-Djavax.jdo.option.ConnectionUserName=hive
-Djavax.jdo.option.ConnectionPassword=" \
 --mount source=warehouse,target=/opt/hive/data/warehouse \
 --name metastore-standalone apache/hive:${HIVE_VERSION} /bin/bash

This should skip the schema initOrUpgrade process:
https://github.com/apache/hive/blob/5022b85b5f50615f85da07bce42aebd414deb9b0/packaging/src/docker/entrypoint.sh#L24
from the 2nd time you run the container.

Regards,
Akshat


On Fri, Dec 22, 2023 at 11:53 AM Sanjay Gupta  wrote:

> Hi All,
>
> If my metastore schema already exists with correct version so what I
> need to do so it doesn't do init or upgrade when starting metastore
> container
>
> I have tried following command line
>
>
> On MAC environment variables
> export HIVE_VERSION=3.1.3
> and even
> SKIP_SCHEMA_INIT=TRUE
>
> docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore
> --add-host=host.docker.internal:host-gateway \
>  --env SKIP_SCHEMA_INIT=TRUE \
>  --env DB_DRIVER=mysql \
>  --env
> SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
>
>  
> -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> -Djavax.jdo.option.ConnectionUserName=hive
> -Djavax.jdo.option.ConnectionPassword=" \
>  --mount source=warehouse,target=/opt/hive/data/warehouse \
>  --name metastore-standalone apache/hive:${HIVE_VERSION} /bin/bash
>
>
>
> ---
>
> Docker Logs , it still tries to initSchema
>
> docker logs 1c
> + : mysql
> + SKIP_SCHEMA_INIT=false
> + export HIVE_CONF_DIR=/opt/hive/conf
> + HIVE_CONF_DIR=/opt/hive/conf
> + '[' -d '' ']'
> + export 'HADOOP_CLIENT_OPTS= -Xmx1G
> -Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
>
> -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> -Djavax.jdo.option.ConnectionUserName=hive
> -Djavax.jdo.option.ConnectionPassword=hive'
> + HADOOP_CLIENT_OPTS=' -Xmx1G
> -Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
>
> -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> -Djavax.jdo.option.ConnectionUserName=hive
> -Djavax.jdo.option.ConnectionPassword=hive'
> + [[ false == \f\a\l\s\e ]]
> + initialize_hive
> + COMMAND=-initOrUpgradeSchema
> ++ cut -d . -f1
> ++ echo 3.1.3
> + '[' 3 -lt 4 ']'
> + COMMAND=-initSchema
> + /opt/hive/bin/schematool -dbType mysql -initSchema
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
>
> [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>
> [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Metastore connection URL:
> jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> Metastore Connection Driver : com.mysql.cj.jdbc.Driver
> Metastore connection User: hive
> Starting metastore schema initialization to 3.1.0
> Initialization script hive-schema-3.1.0.mysql.sql
>
>
> Error: Table 'ctlgs' already exists (state=42S01,code=1050)
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema
> initialization FAILED! Metastore state would be inconsistent !!
> Underlying cause: java.io.IOException : Schema script failed, errorcode 2
> Use --verbose for detailed stacktrace.
> *** schemaTool failed ***
> [WARN] Failed to create directory: /home/hive/.beeline
> No such file or directory
> + '[' 1 -eq 0 ']'
> + echo 'Schema initialization failed!'
> Schema initialization failed!
> + exit 1
> -
>
>
>
> Docker entrypoint.sh have following code
>
> SKIP_SCHEMA_INIT="${IS_RESUME:-false}"
>
> function initialize_hive {
>   COMMAND="-initOrUpgradeSchema"
>   if [ "$(echo "$HIVE_VER" | cut -d '.' -f1)" -lt "4" ]; then
>  COMMAND="-${SCHEMA_COMMAND:-initSchema}"
>   fi
>   $HIVE_HOME/bin/schematool -dbType $DB_DRIVER $COMMAND
>   if [ $? -eq 0 ]; then
> echo "Initialized schema successfully.."
>   else
> echo "Schema initialization failed!"
> exit 1
>   fi
> }
>
> export HIVE_CONF_DIR=$HIVE_HOME/conf
> if [ -d "${HIVE_CUSTOM_CONF_DIR:-}" ]; then
>   find "${HIVE_CUSTOM_CONF_DIR}" -type f -exec \
> ln -sfn {} "${HIVE_CONF_DIR}"/ \;
>   export HADOOP_CONF_DIR=$HIVE_CONF_DIR

How to use SKIP_SCHEMA_INIT=TRUE from command line

2023-12-21 Thread Sanjay Gupta 
Hi All,

If my metastore schema already exists with correct version so what I
need to do so it doesn't do init or upgrade when starting metastore
container

I have tried following command line


On MAC environment variables
export HIVE_VERSION=3.1.3
and even
SKIP_SCHEMA_INIT=TRUE

docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore
--add-host=host.docker.internal:host-gateway \
 --env SKIP_SCHEMA_INIT=TRUE \
 --env DB_DRIVER=mysql \
 --env 
SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
 
-Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
-Djavax.jdo.option.ConnectionUserName=hive
-Djavax.jdo.option.ConnectionPassword=" \
 --mount source=warehouse,target=/opt/hive/data/warehouse \
 --name metastore-standalone apache/hive:${HIVE_VERSION} /bin/bash



---

Docker Logs , it still tries to initSchema

docker logs 1c
+ : mysql
+ SKIP_SCHEMA_INIT=false
+ export HIVE_CONF_DIR=/opt/hive/conf
+ HIVE_CONF_DIR=/opt/hive/conf
+ '[' -d '' ']'
+ export 'HADOOP_CLIENT_OPTS= -Xmx1G
-Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
-Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
-Djavax.jdo.option.ConnectionUserName=hive
-Djavax.jdo.option.ConnectionPassword=hive'
+ HADOOP_CLIENT_OPTS=' -Xmx1G
-Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
-Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
-Djavax.jdo.option.ConnectionUserName=hive
-Djavax.jdo.option.ConnectionPassword=hive'
+ [[ false == \f\a\l\s\e ]]
+ initialize_hive
+ COMMAND=-initOrUpgradeSchema
++ cut -d . -f1
++ echo 3.1.3
+ '[' 3 -lt 4 ']'
+ COMMAND=-initSchema
+ /opt/hive/bin/schematool -dbType mysql -initSchema
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/opt/hive/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:
jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
Metastore Connection Driver : com.mysql.cj.jdbc.Driver
Metastore connection User: hive
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.mysql.sql


Error: Table 'ctlgs' already exists (state=42S01,code=1050)
org.apache.hadoop.hive.metastore.HiveMetaException: Schema
initialization FAILED! Metastore state would be inconsistent !!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
Use --verbose for detailed stacktrace.
*** schemaTool failed ***
[WARN] Failed to create directory: /home/hive/.beeline
No such file or directory
+ '[' 1 -eq 0 ']'
+ echo 'Schema initialization failed!'
Schema initialization failed!
+ exit 1
-



Docker entrypoint.sh have following code

SKIP_SCHEMA_INIT="${IS_RESUME:-false}"

function initialize_hive {
  COMMAND="-initOrUpgradeSchema"
  if [ "$(echo "$HIVE_VER" | cut -d '.' -f1)" -lt "4" ]; then
 COMMAND="-${SCHEMA_COMMAND:-initSchema}"
  fi
  $HIVE_HOME/bin/schematool -dbType $DB_DRIVER $COMMAND
  if [ $? -eq 0 ]; then
echo "Initialized schema successfully.."
  else
echo "Schema initialization failed!"
exit 1
  fi
}

export HIVE_CONF_DIR=$HIVE_HOME/conf
if [ -d "${HIVE_CUSTOM_CONF_DIR:-}" ]; then
  find "${HIVE_CUSTOM_CONF_DIR}" -type f -exec \
ln -sfn {} "${HIVE_CONF_DIR}"/ \;
  export HADOOP_CONF_DIR=$HIVE_CONF_DIR
  export TEZ_CONF_DIR=$HIVE_CONF_DIR
fi

export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Xmx1G $SERVICE_OPTS"
if [[ "${SKIP_SCHEMA_INIT}" == "false" ]]; then
  # handles schema initialization
  initialize_hive
fi

\