[jira] [Created] (HDFS-15489) Documentation link is broken for Apache Hadoop
Namit Maheshwari created HDFS-15489: --- Summary: Documentation link is broken for Apache Hadoop Key: HDFS-15489 URL: https://issues.apache.org/jira/browse/HDFS-15489 Project: Hadoop HDFS Issue Type: Bug Components: documentation Reporter: Namit Maheshwari Attachments: DocumentLinkBroken.mov Please see attached video and screenshots [^DocumentLinkBroken.mov] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2229) ozonefs paths need host and port information for non HA environment
Namit Maheshwari created HDDS-2229: -- Summary: ozonefs paths need host and port information for non HA environment Key: HDDS-2229 URL: https://issues.apache.org/jira/browse/HDDS-2229 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Namit Maheshwari For non HA environment ozonefs path need to have host and port info, like below: o3fs://bucket.volume.om-host:port/key Whereas, for HA environments the path will change to use nameservice like below: o3fs://bucket.volume.ns1/key This will create ambiguity. User experience should be the same irrespective of the usage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-688) Hive Query hangs, if DN's are restarted before the query is submitted
[
https://issues.apache.org/jira/browse/HDDS-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Namit Maheshwari resolved HDDS-688.
---
Resolution: Fixed
This is fixed with the recent changes. Resolving it.
> Hive Query hangs, if DN's are restarted before the query is submitted
> -
>
> Key: HDDS-688
> URL: https://issues.apache.org/jira/browse/HDDS-688
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Namit Maheshwari
>Assignee: Mukul Kumar Singh
>Priority: Major
>
> Run a Hive Insert Query. It runs fine as below:
> {code:java}
> 0: jdbc:hive2://ctr-e138-1518143905142-510793> insert into testo3 values(1,
> "aa", 3.0);
> INFO : Compiling
> command(queryId=hive_20181018005729_fe644ab2-f8cc-41c3-b2d8-ffe1022de607):
> insert into testo3 values(1, "aa", 3.0)
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_col0,
> type:int, comment:null), FieldSchema(name:_col1, type:string, comment:null),
> FieldSchema(name:_col2, type:float, comment:null)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20181018005729_fe644ab2-f8cc-41c3-b2d8-ffe1022de607);
> Time taken: 0.52 seconds
> INFO : Executing
> command(queryId=hive_20181018005729_fe644ab2-f8cc-41c3-b2d8-ffe1022de607):
> insert into testo3 values(1, "aa", 3.0)
> INFO : Query ID = hive_20181018005729_fe644ab2-f8cc-41c3-b2d8-ffe1022de607
> INFO : Total jobs = 1
> INFO : Launching Job 1 out of 1
> INFO : Starting task [Stage-1:MAPRED] in serial mode
> INFO : Subscribed to counters: [] for queryId:
> hive_20181018005729_fe644ab2-f8cc-41c3-b2d8-ffe1022de607
> INFO : Session is already open
> INFO : Dag name: insert into testo3 values(1, "aa", 3.0) (Stage-1)
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1539383731490_0073)
> --
> VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
> --
> Map 1 .. container SUCCEEDED 1 1 0 0 0 0
> Reducer 2 .. container SUCCEEDED 1 1 0 0 0 0
> --
> VERTICES: 02/02 [==>>] 100% ELAPSED TIME: 11.95 s
> --
> INFO : Status: DAG finished successfully in 10.68 seconds
> INFO :
> INFO : Query Execution Summary
> INFO :
> --
> INFO : OPERATION DURATION
> INFO :
> --
> INFO : Compile Query 0.52s
> INFO : Prepare Plan 0.23s
> INFO : Get Query Coordinator (AM) 0.00s
> INFO : Submit Plan 0.11s
> INFO : Start DAG 0.57s
> INFO : Run DAG 10.68s
> INFO :
> --
> INFO :
> INFO : Task Execution Summary
> INFO :
> --
> INFO : VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> INFO :
> --
> INFO : Map 1 7074.00 11,280 276 3 1
> INFO : Reducer 2 1074.00 2,040 0 1 0
> INFO :
> --
> INFO :
> INFO : org.apache.tez.common.counters.DAGCounter:
> INFO : NUM_SUCCEEDED_TASKS: 2
> INFO : TOTAL_LAUNCHED_TASKS: 2
> INFO : AM_CPU_MILLISECONDS: 1390
> INFO : AM_GC_TIME_MILLIS: 0
> INFO : File System Counters:
> INFO : FILE_BYTES_READ: 135
> INFO : FILE_BYTES_WRITTEN: 135
> INFO : HDFS_BYTES_WRITTEN: 199
> INFO : HDFS_READ_OPS: 3
> INFO : HDFS_WRITE_OPS: 2
> INFO : HDFS_OP_CREATE: 1
> INFO : HDFS_OP_GET_FILE_STATUS: 3
> INFO : HDFS_OP_RENAME: 1
> INFO : org.apache.tez.common.counters.TaskCounter:
> INFO : SPILLED_RECORDS: 0
> INFO : NUM_SHUFFLED_INPUTS: 1
> INFO : NUM_FAILED_SHUFFLE_INPUTS: 0
> INFO : GC_TIME_MILLIS: 276
> INFO : TASK_DURATION_MILLIS: 8474
> INFO : CPU_MILLISECONDS: 13320
> INFO : PHYSICAL_MEMORY_BYTES: 4294967296
> INFO : VIRTUAL_MEMORY_BYTES: 11205029888
> INFO : COMMITTED_HEAP_BYTES: 4294967296
> INFO : INPUT_RECORDS_PROCESSED: 5
> INFO : INPUT_SPLIT_LENGTH_BYTES: 1
> INFO : OUTPUT_RECORDS: 1
> INFO : OUTPUT_LARGE_RECORDS: 0
> INFO : OUTPUT_BYTES: 94
> INFO : OUTPUT_BYTES_WITH_OVERHEAD: 102
> INFO : OUTPUT_BYTES_PHYSICAL: 127
> INFO : ADDITIONAL_SPILLS_BYTES_WRITTEN: 0
> INFO :
[jira] [Created] (HDDS-689) Datanode shuts down on restart
Namit Maheshwari created HDDS-689:
-
Summary: Datanode shuts down on restart
Key: HDDS-689
URL: https://issues.apache.org/jira/browse/HDDS-689
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
Restart all the 8 DNs in the cluster. 2 of them shuts down as below
{code:java}
2018-10-18 01:10:57,102 ERROR impl.StateMachineUpdater
(ExitUtils.java:terminate(86)) - Terminating with exit status 2:
StateMachineUpdater-69d15283-4e2e-4c30-a028-f2bad0f83cc1: the
StateMachineUpdater hits Throwable
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.CompletableFuture$AsyncSupply@7ee4907b rejected from
java.util.concurrent.ThreadPoolExecutor@702371d7[Terminated, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 0]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at
java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668)
at
java.util.concurrent.CompletableFuture.asyncSupplyStage(CompletableFuture.java:1604)
at
java.util.concurrent.CompletableFuture.supplyAsync(CompletableFuture.java:1830)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.applyTransaction(ContainerStateMachine.java:433)
at
org.apache.ratis.server.impl.RaftServerImpl.applyLogToStateMachine(RaftServerImpl.java:1093)
at
org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:148)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:57,107 WARN fs.CachingGetSpaceUsed
(CachingGetSpaceUsed.java:run(183)) - Thread Interrupted waiting to refresh
disk information: sleep interrupted
2018-10-18 01:10:57,108 INFO datanode.DataNode (LogAdapter.java:info(51)) -
SHUTDOWN_MSG:
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-688) Hive Query hangs, if DN's are restarted before the query is submitted
Namit Maheshwari created HDDS-688:
-
Summary: Hive Query hangs, if DN's are restarted before the query
is submitted
Key: HDDS-688
URL: https://issues.apache.org/jira/browse/HDDS-688
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
Run a Hive Insert Query. It runs fine as below:
{code:java}
0: jdbc:hive2://ctr-e138-1518143905142-510793> insert into testo3 values(1,
"aa", 3.0);
INFO : Compiling
command(queryId=hive_20181018005729_fe644ab2-f8cc-41c3-b2d8-ffe1022de607):
insert into testo3 values(1, "aa", 3.0)
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_col0,
type:int, comment:null), FieldSchema(name:_col1, type:string, comment:null),
FieldSchema(name:_col2, type:float, comment:null)], properties:null)
INFO : Completed compiling
command(queryId=hive_20181018005729_fe644ab2-f8cc-41c3-b2d8-ffe1022de607); Time
taken: 0.52 seconds
INFO : Executing
command(queryId=hive_20181018005729_fe644ab2-f8cc-41c3-b2d8-ffe1022de607):
insert into testo3 values(1, "aa", 3.0)
INFO : Query ID = hive_20181018005729_fe644ab2-f8cc-41c3-b2d8-ffe1022de607
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : Subscribed to counters: [] for queryId:
hive_20181018005729_fe644ab2-f8cc-41c3-b2d8-ffe1022de607
INFO : Session is already open
INFO : Dag name: insert into testo3 values(1, "aa", 3.0) (Stage-1)
INFO : Status: Running (Executing on YARN cluster with App id
application_1539383731490_0073)
--
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--
Map 1 .. container SUCCEEDED 1 1 0 0 0 0
Reducer 2 .. container SUCCEEDED 1 1 0 0 0 0
--
VERTICES: 02/02 [==>>] 100% ELAPSED TIME: 11.95 s
--
INFO : Status: DAG finished successfully in 10.68 seconds
INFO :
INFO : Query Execution Summary
INFO :
--
INFO : OPERATION DURATION
INFO :
--
INFO : Compile Query 0.52s
INFO : Prepare Plan 0.23s
INFO : Get Query Coordinator (AM) 0.00s
INFO : Submit Plan 0.11s
INFO : Start DAG 0.57s
INFO : Run DAG 10.68s
INFO :
--
INFO :
INFO : Task Execution Summary
INFO :
--
INFO : VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
OUTPUT_RECORDS
INFO :
--
INFO : Map 1 7074.00 11,280 276 3 1
INFO : Reducer 2 1074.00 2,040 0 1 0
INFO :
--
INFO :
INFO : org.apache.tez.common.counters.DAGCounter:
INFO : NUM_SUCCEEDED_TASKS: 2
INFO : TOTAL_LAUNCHED_TASKS: 2
INFO : AM_CPU_MILLISECONDS: 1390
INFO : AM_GC_TIME_MILLIS: 0
INFO : File System Counters:
INFO : FILE_BYTES_READ: 135
INFO : FILE_BYTES_WRITTEN: 135
INFO : HDFS_BYTES_WRITTEN: 199
INFO : HDFS_READ_OPS: 3
INFO : HDFS_WRITE_OPS: 2
INFO : HDFS_OP_CREATE: 1
INFO : HDFS_OP_GET_FILE_STATUS: 3
INFO : HDFS_OP_RENAME: 1
INFO : org.apache.tez.common.counters.TaskCounter:
INFO : SPILLED_RECORDS: 0
INFO : NUM_SHUFFLED_INPUTS: 1
INFO : NUM_FAILED_SHUFFLE_INPUTS: 0
INFO : GC_TIME_MILLIS: 276
INFO : TASK_DURATION_MILLIS: 8474
INFO : CPU_MILLISECONDS: 13320
INFO : PHYSICAL_MEMORY_BYTES: 4294967296
INFO : VIRTUAL_MEMORY_BYTES: 11205029888
INFO : COMMITTED_HEAP_BYTES: 4294967296
INFO : INPUT_RECORDS_PROCESSED: 5
INFO : INPUT_SPLIT_LENGTH_BYTES: 1
INFO : OUTPUT_RECORDS: 1
INFO : OUTPUT_LARGE_RECORDS: 0
INFO : OUTPUT_BYTES: 94
INFO : OUTPUT_BYTES_WITH_OVERHEAD: 102
INFO : OUTPUT_BYTES_PHYSICAL: 127
INFO : ADDITIONAL_SPILLS_BYTES_WRITTEN: 0
INFO : ADDITIONAL_SPILLS_BYTES_READ: 0
INFO : ADDITIONAL_SPILL_COUNT: 0
INFO : SHUFFLE_BYTES: 103
INFO : SHUFFLE_BYTES_DECOMPRESSED: 102
INFO : SHUFFLE_BYTES_TO_MEM: 0
INFO : SHUFFLE_BYTES_TO_DISK: 0
INFO : SHUFFLE_BYTES_DISK_DIRECT: 103
INFO : SHUFFLE_PHASE_TIME: 154
INFO : FIRST_EVENT_RECEIVED: 108
INFO : LAST_EVENT_RECEIVED: 108
INFO : HIVE:
INFO : CREATED_FILES: 2
INFO : DESERIALIZE_ERRORS: 0
INFO : RECORDS_IN_Map_1: 3
INFO : RECORDS_OUT_0: 1
INFO : RECORDS_OUT_1_default.testo3: 1
INFO : RECORDS_OUT_INTERMEDIATE_Map_1: 1
INFO :
[jira] [Created] (HDDS-687) SCM is not able to restart
Namit Maheshwari created HDDS-687: - Summary: SCM is not able to restart Key: HDDS-687 URL: https://issues.apache.org/jira/browse/HDDS-687 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Namit Maheshwari SCM is not able to come up on restart. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-674) Not able to get key after distcp job passes
Namit Maheshwari created HDDS-674:
-
Summary: Not able to get key after distcp job passes
Key: HDDS-674
URL: https://issues.apache.org/jira/browse/HDDS-674
Project: Hadoop Distributed Data Store
Issue Type: Task
Reporter: Namit Maheshwari
It fails with
{code:java}
-bash-4.2$ ozone sh key get /volume2/bucket2/distcp/wordcount_input_1.txt
/tmp/wordcountDistcp.txt
2018-10-17 00:25:07,904 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Lookup key failed, error:KEY_NOT_FOUND{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-672) Spark shell throws OzoneFileSystem not found
Namit Maheshwari created HDDS-672: - Summary: Spark shell throws OzoneFileSystem not found Key: HDDS-672 URL: https://issues.apache.org/jira/browse/HDDS-672 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Namit Maheshwari Spark shell throws OzoneFileSystem not found, if the ozone jars are not specified in the --jars options -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-671) Hive HSI insert tries to create data in Hdfs for Ozone external table
Namit Maheshwari created HDDS-671: - Summary: Hive HSI insert tries to create data in Hdfs for Ozone external table Key: HDDS-671 URL: https://issues.apache.org/jira/browse/HDDS-671 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Namit Maheshwari Hive HSI insert tries to create data in Hdfs for Ozone external table, when "hive.server2.enable.doAs" is set to true -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-670) Hive insert fails against Ozone external table
Namit Maheshwari created HDDS-670:
-
Summary: Hive insert fails against Ozone external table
Key: HDDS-670
URL: https://issues.apache.org/jira/browse/HDDS-670
Project: Hadoop Distributed Data Store
Issue Type: Task
Reporter: Namit Maheshwari
It fails with
{code:java}
ERROR : Job Commit failed with exception
'org.apache.hadoop.hive.ql.metadata.HiveException(Unable to move:
o3://bucket2.volume2/testo3/.hive-staging_hive_2018-10-16_21-09-35_130_1001829123585250245-1/_tmp.-ext-1
to:
o3://bucket2.volume2/testo3/.hive-staging_hive_2018-10-16_21-09-35_130_1001829123585250245-1/_tmp.-ext-1.moved)'
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move:
o3://bucket2.volume2/testo3/.hive-staging_hive_2018-10-16_21-09-35_130_1001829123585250245-1/_tmp.-ext-1
to:
o3://bucket2.volume2/testo3/.hive-staging_hive_2018-10-16_21-09-35_130_1001829123585250245-1/_tmp.-ext-1.moved
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-664) Creating hive table on Ozone fails
Namit Maheshwari created HDDS-664:
-
Summary: Creating hive table on Ozone fails
Key: HDDS-664
URL: https://issues.apache.org/jira/browse/HDDS-664
Project: Hadoop Distributed Data Store
Issue Type: Task
Reporter: Namit Maheshwari
Modified HIVE_AUX_JARS_PATH to include Ozone jars. Tried creating Hive external
table on Ozone. It fails with "Error: Error while compiling statement: FAILED:
HiveAuthzPluginException Error getting permissions for
o3://bucket2.volume2/testo3: User: hive is not allowed to impersonate anonymous
(state=42000,code=4)"
{code:java}
-bash-4.2$ beeline
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/hdp/3.0.3.0-63/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/hdp/3.0.3.0-63/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to
jdbc:hive2://ctr-e138-1518143905142-510793-01-11.hwx.site:2181,ctr-e138-1518143905142-510793-01-06.hwx.site:2181,ctr-e138-1518143905142-510793-01-08.hwx.site:2181,ctr-e138-1518143905142-510793-01-10.hwx.site:2181,ctr-e138-1518143905142-510793-01-07.hwx.site:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
Enter username for
jdbc:hive2://ctr-e138-1518143905142-510793-01-11.hwx.site:2181,ctr-e138-1518143905142-510793-01-06.hwx.site:2181,ctr-e138-1518143905142-510793-01-08.hwx.site:2181,ctr-e138-1518143905142-510793-01-10.hwx.site:2181,ctr-e138-1518143905142-510793-01-07.hwx.site:2181/default:
Enter password for
jdbc:hive2://ctr-e138-1518143905142-510793-01-11.hwx.site:2181,ctr-e138-1518143905142-510793-01-06.hwx.site:2181,ctr-e138-1518143905142-510793-01-08.hwx.site:2181,ctr-e138-1518143905142-510793-01-10.hwx.site:2181,ctr-e138-1518143905142-510793-01-07.hwx.site:2181/default:
18/10/15 21:36:55 [main]: INFO jdbc.HiveConnection: Connected to
ctr-e138-1518143905142-510793-01-04.hwx.site:1
Connected to: Apache Hive (version 3.1.0.3.0.3.0-63)
Driver: Hive JDBC (version 3.1.0.3.0.3.0-63)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 3.1.0.3.0.3.0-63 by Apache Hive
0: jdbc:hive2://ctr-e138-1518143905142-510793> create external table testo3 ( i
int, s string, d float) location "o3://bucket2.volume2/testo3";
Error: Error while compiling statement: FAILED: HiveAuthzPluginException Error
getting permissions for o3://bucket2.volume2/testo3: User: hive is not allowed
to impersonate anonymous (state=42000,code=4)
0: jdbc:hive2://ctr-e138-1518143905142-510793> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-637) Not able to access the part-r-00000 file after the MR job succeeds
[
https://issues.apache.org/jira/browse/HDDS-637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Namit Maheshwari resolved HDDS-637.
---
Resolution: Cannot Reproduce
The issue seem to go away with the latest code.
Will reopen if seen again. Thanks [~xyao]
> Not able to access the part-r-0 file after the MR job succeeds
> --
>
> Key: HDDS-637
> URL: https://issues.apache.org/jira/browse/HDDS-637
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Namit Maheshwari
>Assignee: Xiaoyu Yao
>Priority: Major
>
> Run a MR job
> {code:java}
> -bash-4.2$ /usr/hdp/current/hadoop-client/bin/hadoop jar
> /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar
> wordcount /tmp/mr_jobs/input/ o3://bucket2.volume2/mr_jobDD
> 18/10/12 01:00:23 INFO conf.Configuration: Removed undeclared tags:
> 18/10/12 01:00:24 INFO conf.Configuration: Removed undeclared tags:
> 18/10/12 01:00:24 INFO conf.Configuration: Removed undeclared tags:
> 18/10/12 01:00:25 INFO conf.Configuration: Removed undeclared tags:
> 18/10/12 01:00:25 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
> to rm2
> 18/10/12 01:00:25 INFO mapreduce.JobResourceUploader: Disabling Erasure
> Coding for path: /user/hdfs/.staging/job_1539295307098_0003
> 18/10/12 01:00:27 INFO input.FileInputFormat: Total input files to process : 1
> 18/10/12 01:00:27 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
> 18/10/12 01:00:27 INFO lzo.LzoCodec: Successfully loaded & initialized
> native-lzo library [hadoop-lzo rev 5d6248d8d690f8456469979213ab2e9993bfa2e9]
> 18/10/12 01:00:27 INFO mapreduce.JobSubmitter: number of splits:1
> 18/10/12 01:00:28 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1539295307098_0003
> 18/10/12 01:00:28 INFO mapreduce.JobSubmitter: Executing with tokens: []
> 18/10/12 01:00:28 INFO conf.Configuration: Removed undeclared tags:
> 18/10/12 01:00:28 INFO conf.Configuration: found resource resource-types.xml
> at file:/etc/hadoop/3.0.3.0-63/0/resource-types.xml
> 18/10/12 01:00:28 INFO conf.Configuration: Removed undeclared tags:
> 18/10/12 01:00:28 INFO impl.YarnClientImpl: Submitted application
> application_1539295307098_0003
> 18/10/12 01:00:29 INFO mapreduce.Job: The url to track the job:
> http://ctr-e138-1518143905142-510793-01-05.hwx.site:8088/proxy/application_1539295307098_0003/
> 18/10/12 01:00:29 INFO mapreduce.Job: Running job: job_1539295307098_0003
> 18/10/12 01:00:35 INFO mapreduce.Job: Job job_1539295307098_0003 running in
> uber mode : false
> 18/10/12 01:00:35 INFO mapreduce.Job: map 0% reduce 0%
> 18/10/12 01:00:44 INFO mapreduce.Job: map 100% reduce 0%
> 18/10/12 01:00:57 INFO mapreduce.Job: map 100% reduce 67%
> 18/10/12 01:00:59 INFO mapreduce.Job: map 100% reduce 100%
> 18/10/12 01:00:59 INFO mapreduce.Job: Job job_1539295307098_0003 completed
> successfully
> 18/10/12 01:00:59 INFO conf.Configuration: Removed undeclared tags:
> 18/10/12 01:00:59 INFO mapreduce.Job: Counters: 58
> File System Counters
> FILE: Number of bytes read=6332
> FILE: Number of bytes written=532585
> FILE: Number of read operations=0
> FILE: Number of large read operations=0
> FILE: Number of write operations=0
> HDFS: Number of bytes read=215876
> HDFS: Number of bytes written=0
> HDFS: Number of read operations=2
> HDFS: Number of large read operations=0
> HDFS: Number of write operations=0
> O3: Number of bytes read=0
> O3: Number of bytes written=0
> O3: Number of read operations=0
> O3: Number of large read operations=0
> O3: Number of write operations=0
> Job Counters
> Launched map tasks=1
> Launched reduce tasks=1
> Rack-local map tasks=1
> Total time spent by all maps in occupied slots (ms)=25392
> Total time spent by all reduces in occupied slots (ms)=103584
> Total time spent by all map tasks (ms)=6348
> Total time spent by all reduce tasks (ms)=12948
> Total vcore-milliseconds taken by all map tasks=6348
> Total vcore-milliseconds taken by all reduce tasks=12948
> Total megabyte-milliseconds taken by all map tasks=26001408
> Total megabyte-milliseconds taken by all reduce tasks=106070016
> Map-Reduce Framework
> Map input records=716
> Map output records=32019
> Map output bytes=343475
> Map output materialized bytes=6332
> Input split bytes=121
> Combine input records=32019
> Combine output records=461
> Reduce input groups=461
> Reduce shuffle bytes=6332
> Reduce input records=461
> Reduce output records=461
> Spilled Records=922
> Shuffled Maps =1
> Failed Shuffles=0
> Merged Map outputs=1
> GC time elapsed (ms)=359
> CPU time spent (ms)=11800
> Physical memory (bytes) snapshot=3018502144
> Virtual memory (bytes) snapshot=14470242304
> Total committed heap usage (bytes)=3521642496
> Peak Map Physical memory
[jira] [Created] (HDDS-663) Lot of "Removed undeclared tags" logger while running commands
Namit Maheshwari created HDDS-663:
-
Summary: Lot of "Removed undeclared tags" logger while running
commands
Key: HDDS-663
URL: https://issues.apache.org/jira/browse/HDDS-663
Project: Hadoop Distributed Data Store
Issue Type: Task
Reporter: Namit Maheshwari
While running commands against OzoneFs see lot of logger like below:
{code:java}
-bash-4.2$ hdfs dfs -ls o3://bucket2.volume2/mr_jobEE
18/10/15 20:29:17 INFO conf.Configuration: Removed undeclared tags:
18/10/15 20:29:18 INFO conf.Configuration: Removed undeclared tags:
Found 2 items
-rw-rw-rw- 1 hdfs hdfs 0 2018-10-15 20:28 o3://bucket2.volume2/mr_jobEE/_SUCCESS
-rw-rw-rw- 1 hdfs hdfs 5017 1970-07-23 04:33
o3://bucket2.volume2/mr_jobEE/part-r-0
18/10/15 20:29:19 INFO conf.Configuration: Removed undeclared tags:
-bash-4.2${code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-652) Properties in ozone-site.xml does not work well with IP
Namit Maheshwari created HDDS-652: - Summary: Properties in ozone-site.xml does not work well with IP Key: HDDS-652 URL: https://issues.apache.org/jira/browse/HDDS-652 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Namit Maheshwari There have been cases where properties in ozone-site.xml does not work well with IP. If those properties like ozone.om.address is changed to use hostnames, it works well. Ideally this should work fine with both IP and hostnames. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-651) Rename o3 to o3fs for Filesystem
Namit Maheshwari created HDDS-651: - Summary: Rename o3 to o3fs for Filesystem Key: HDDS-651 URL: https://issues.apache.org/jira/browse/HDDS-651 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Namit Maheshwari I propose that we rename o3 to o3fs for Filesystem. It creates a lot of confusion while using the same name o3 for different purposes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-650) Spark job is not able to pick up Ozone configuration
Namit Maheshwari created HDDS-650:
-
Summary: Spark job is not able to pick up Ozone configuration
Key: HDDS-650
URL: https://issues.apache.org/jira/browse/HDDS-650
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
Spark job is not able to pick up Ozone configuration.
{code:java}
-bash-4.2$ spark-shell --master yarn-client --jars
/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.3.0.3.0-63.jar,/tmp/ozone-0.3.0-SNAPSHOT/share/hadoop/ozoneplugin/hadoop-ozone-datanode-plugin-0.3.0-SNAPSHOT.jar,/tmp/ozone-0.3.0-SNAPSHOT/share/hadoop/ozonefs/hadoop-ozone-filesystem-0.3.0-SNAPSHOT.jar
Warning: Master yarn-client is deprecated since 2.0. Please use master "yarn"
with specified deploy mode instead.
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
Spark context Web UI available at
http://ctr-e138-1518143905142-510793-01-02.hwx.site:4040
Spark context available as 'sc' (master = yarn, app id =
application_1539295307098_0011).
Spark session available as 'spark'.
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.3.2.3.0.3.0-63
/_/
Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_181)
Type in expressions to have them evaluated.
Type :help for more information.
scala>
scala> val input = sc.textFile("o3://bucket2.volume2/passwd");
input: org.apache.spark.rdd.RDD[String] = o3://bucket2.volume2/passwd
MapPartitionsRDD[1] at textFile at :24
scala> val count = input.flatMap(line => line.split(" ")).map(word => (word,
1)).reduceByKey(_+_);
count: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[4] at reduceByKey
at :25
scala> count.cache()
res0: count.type = ShuffledRDD[4] at reduceByKey at :25
scala> count.saveAsTextFile("o3://bucket2.volume2/sparkout3");
[Stage 0:> (0 + 2) / 2]18/10/12 22:16:44 WARN TaskSetManager: Lost task 1.0 in
stage 0.0 (TID 1, ctr-e138-1518143905142-510793-01-11.hwx.site, executor
1): java.io.IOException: Couldn't create protocol class
org.apache.hadoop.ozone.client.rpc.RpcClient
at
org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:299)
at
org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:169)
at
org.apache.hadoop.fs.ozone.OzoneFileSystem.initialize(OzoneFileSystem.java:119)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
at org.apache.hadoop.mapred.LineRecordReader.(LineRecordReader.java:108)
at
org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
at org.apache.spark.rdd.HadoopRDD$$anon$1.liftedTree1$1(HadoopRDD.scala:257)
at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:256)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:214)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:94)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: ozone.om.address must be
defined. See https://wiki.apache.org/hadoop/Ozone#Configuration for details on
configuring Ozone.
at org.apache.hadoop.ozone.OmUtils.getOmAddressForClients(OmUtils.java:70)
at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:114)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
[jira] [Created] (HDDS-637) Not able to access the part-r-00000 file after the MR job succeeds
Namit Maheshwari created HDDS-637:
-
Summary: Not able to access the part-r-0 file after the MR job
succeeds
Key: HDDS-637
URL: https://issues.apache.org/jira/browse/HDDS-637
Project: Hadoop Distributed Data Store
Issue Type: Bug
Affects Versions: 0.3.0
Reporter: Namit Maheshwari
Run a MR job
{code:java}
-bash-4.2$ /usr/hdp/current/hadoop-client/bin/hadoop jar
/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar
wordcount /tmp/mr_jobs/input/ o3://bucket2.volume2/mr_jobDD
18/10/12 01:00:23 INFO conf.Configuration: Removed undeclared tags:
18/10/12 01:00:24 INFO conf.Configuration: Removed undeclared tags:
18/10/12 01:00:24 INFO conf.Configuration: Removed undeclared tags:
18/10/12 01:00:25 INFO conf.Configuration: Removed undeclared tags:
18/10/12 01:00:25 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm2
18/10/12 01:00:25 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding
for path: /user/hdfs/.staging/job_1539295307098_0003
18/10/12 01:00:27 INFO input.FileInputFormat: Total input files to process : 1
18/10/12 01:00:27 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
18/10/12 01:00:27 INFO lzo.LzoCodec: Successfully loaded & initialized
native-lzo library [hadoop-lzo rev 5d6248d8d690f8456469979213ab2e9993bfa2e9]
18/10/12 01:00:27 INFO mapreduce.JobSubmitter: number of splits:1
18/10/12 01:00:28 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1539295307098_0003
18/10/12 01:00:28 INFO mapreduce.JobSubmitter: Executing with tokens: []
18/10/12 01:00:28 INFO conf.Configuration: Removed undeclared tags:
18/10/12 01:00:28 INFO conf.Configuration: found resource resource-types.xml at
file:/etc/hadoop/3.0.3.0-63/0/resource-types.xml
18/10/12 01:00:28 INFO conf.Configuration: Removed undeclared tags:
18/10/12 01:00:28 INFO impl.YarnClientImpl: Submitted application
application_1539295307098_0003
18/10/12 01:00:29 INFO mapreduce.Job: The url to track the job:
http://ctr-e138-1518143905142-510793-01-05.hwx.site:8088/proxy/application_1539295307098_0003/
18/10/12 01:00:29 INFO mapreduce.Job: Running job: job_1539295307098_0003
18/10/12 01:00:35 INFO mapreduce.Job: Job job_1539295307098_0003 running in
uber mode : false
18/10/12 01:00:35 INFO mapreduce.Job: map 0% reduce 0%
18/10/12 01:00:44 INFO mapreduce.Job: map 100% reduce 0%
18/10/12 01:00:57 INFO mapreduce.Job: map 100% reduce 67%
18/10/12 01:00:59 INFO mapreduce.Job: map 100% reduce 100%
18/10/12 01:00:59 INFO mapreduce.Job: Job job_1539295307098_0003 completed
successfully
18/10/12 01:00:59 INFO conf.Configuration: Removed undeclared tags:
18/10/12 01:00:59 INFO mapreduce.Job: Counters: 58
File System Counters
FILE: Number of bytes read=6332
FILE: Number of bytes written=532585
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=215876
HDFS: Number of bytes written=0
HDFS: Number of read operations=2
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
O3: Number of bytes read=0
O3: Number of bytes written=0
O3: Number of read operations=0
O3: Number of large read operations=0
O3: Number of write operations=0
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Rack-local map tasks=1
Total time spent by all maps in occupied slots (ms)=25392
Total time spent by all reduces in occupied slots (ms)=103584
Total time spent by all map tasks (ms)=6348
Total time spent by all reduce tasks (ms)=12948
Total vcore-milliseconds taken by all map tasks=6348
Total vcore-milliseconds taken by all reduce tasks=12948
Total megabyte-milliseconds taken by all map tasks=26001408
Total megabyte-milliseconds taken by all reduce tasks=106070016
Map-Reduce Framework
Map input records=716
Map output records=32019
Map output bytes=343475
Map output materialized bytes=6332
Input split bytes=121
Combine input records=32019
Combine output records=461
Reduce input groups=461
Reduce shuffle bytes=6332
Reduce input records=461
Reduce output records=461
Spilled Records=922
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=359
CPU time spent (ms)=11800
Physical memory (bytes) snapshot=3018502144
Virtual memory (bytes) snapshot=14470242304
Total committed heap usage (bytes)=3521642496
Peak Map Physical memory (bytes)=2518896640
Peak Map Virtual memory (bytes)=5397549056
Peak Reduce Physical memory (bytes)=499605504
Peak Reduce Virtual memory (bytes)=9072693248
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=215755
File Output Format Counters
Bytes Written=0
18/10/12 01:00:59 INFO conf.Configuration: Removed undeclared tags:
-bash-4.2$
{code}
Below exception is seen in SCM logs
{code:java}
2018-10-12 01:00:51,142 INFO
[jira] [Created] (HDDS-624) PutBlock fails with Unexpected Storage Container Exception
Namit Maheshwari created HDDS-624:
-
Summary: PutBlock fails with Unexpected Storage Container Exception
Key: HDDS-624
URL: https://issues.apache.org/jira/browse/HDDS-624
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
As per HDDS-622, Datanodes were shutting down while running MR jobs due to
issue in RocksDBStore. To avoid that failure set the property
_ozone.metastore.rocksdb.statistics_ to _OFF_ in ozone-site.xml
Now running Mapreduce job fails with below error
{code:java}
-bash-4.2$ /usr/hdp/current/hadoop-client/bin/hadoop jar
/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar
wordcount /tmp/mr_jobs/input/ o3://bucket2.volume2/mr_jobAA
18/10/11 00:14:41 INFO conf.Configuration: Removed undeclared tags:
18/10/11 00:14:42 INFO conf.Configuration: Removed undeclared tags:
18/10/11 00:14:42 INFO conf.Configuration: Removed undeclared tags:
18/10/11 00:14:43 INFO conf.Configuration: Removed undeclared tags:
18/10/11 00:14:43 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm2
18/10/11 00:14:43 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding
for path: /user/hdfs/.staging/job_1539208750583_0005
18/10/11 00:14:43 INFO input.FileInputFormat: Total input files to process : 1
18/10/11 00:14:43 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
18/10/11 00:14:43 INFO lzo.LzoCodec: Successfully loaded & initialized
native-lzo library [hadoop-lzo rev 5d6248d8d690f8456469979213ab2e9993bfa2e9]
18/10/11 00:14:44 INFO mapreduce.JobSubmitter: number of splits:1
18/10/11 00:14:44 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1539208750583_0005
18/10/11 00:14:44 INFO mapreduce.JobSubmitter: Executing with tokens: []
18/10/11 00:14:44 INFO conf.Configuration: Removed undeclared tags:
18/10/11 00:14:44 INFO conf.Configuration: found resource resource-types.xml at
file:/etc/hadoop/3.0.3.0-63/0/resource-types.xml
18/10/11 00:14:44 INFO conf.Configuration: Removed undeclared tags:
18/10/11 00:14:44 INFO impl.YarnClientImpl: Submitted application
application_1539208750583_0005
18/10/11 00:14:45 INFO mapreduce.Job: The url to track the job:
http://ctr-e138-1518143905142-510793-01-05.hwx.site:8088/proxy/application_1539208750583_0005/
18/10/11 00:14:45 INFO mapreduce.Job: Running job: job_1539208750583_0005
18/10/11 00:14:53 INFO mapreduce.Job: Job job_1539208750583_0005 running in
uber mode : false
18/10/11 00:14:53 INFO mapreduce.Job: map 0% reduce 0%
18/10/11 00:15:00 INFO mapreduce.Job: map 100% reduce 0%
18/10/11 00:15:10 INFO mapreduce.Job: map 100% reduce 67%
18/10/11 00:15:11 INFO mapreduce.Job: Task Id :
attempt_1539208750583_0005_r_00_0, Status : FAILED
Error: java.io.IOException: Unexpected Storage Container Exception:
java.io.IOException: Failed to command cmdType: PutBlock
traceID: "df0ed956-fa4d-40ef-a7f2-ec0b6160b41b"
containerID: 2
datanodeUuid: "96f8fa78-413e-4350-a8ff-6cbdaa16ba7f"
putBlock {
blockData {
blockID {
containerID: 2
localID: 100874119214399488
}
metadata {
key: "TYPE"
value: "KEY"
}
chunks {
chunkName:
"f24fa36171bda3113584cb01dc12a871_stream_84157b3a-654d-4e3d-8455-fbf85321a306_chunk_1"
offset: 0
len: 5017
}
}
}
at
org.apache.hadoop.hdds.scm.storage.ChunkOutputStream.close(ChunkOutputStream.java:171)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream$ChunkOutputStreamEntry.close(ChunkGroupOutputStream.java:699)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:502)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.close(ChunkGroupOutputStream.java:531)
at
org.apache.hadoop.fs.ozone.OzoneFSOutputStream.close(OzoneFSOutputStream.java:57)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at
org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106)
at
org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close(ReduceTask.java:551)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:630)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.io.IOException: Failed to command cmdType: PutBlock
traceID: "df0ed956-fa4d-40ef-a7f2-ec0b6160b41b"
containerID: 2
datanodeUuid: "96f8fa78-413e-4350-a8ff-6cbdaa16ba7f"
putBlock {
blockData {
blockID {
containerID: 2
localID: 100874119214399488
}
metadata {
key: "TYPE"
value: "KEY"
}
chunks {
chunkName:
[jira] [Created] (HDDS-623) On SCM UI, Node Manager info is empty
Namit Maheshwari created HDDS-623: - Summary: On SCM UI, Node Manager info is empty Key: HDDS-623 URL: https://issues.apache.org/jira/browse/HDDS-623 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Namit Maheshwari Attachments: Screen Shot 2018-10-10 at 4.19.59 PM.png Fields like below are empty Node Manager: Minimum chill mode nodes Node Manager: Out-of-node chill mode Node Manager: Chill mode status Node Manager: Manual chill mode Please see attached screenshot !Screen Shot 2018-10-10 at 4.19.59 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-622) Datanode shuts down with RocksDBStore java.lang.NoSuchMethodError
Namit Maheshwari created HDDS-622:
-
Summary: Datanode shuts down with RocksDBStore
java.lang.NoSuchMethodError
Key: HDDS-622
URL: https://issues.apache.org/jira/browse/HDDS-622
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
Datanodes are registered fine on a Hadoop + Ozone cluster.
While running jobs against Ozone, Datanode shuts down as below:
{code:java}
2018-10-10 21:50:42,708 INFO storage.RaftLogWorker
(RaftLogWorker.java:rollLogSegment(263)) - Rolling
segment:7c1a32b5-34ed-4a2a-aa07-ac75d25858b6-RaftLogWorker index to:2
2018-10-10 21:50:42,714 INFO impl.RaftServerImpl
(ServerState.java:setRaftConf(319)) - 7c1a32b5-34ed-4a2a-aa07-ac75d25858b6: set
configuration 2: [7c1a32b5-34ed-4a2a-aa07-ac75d25858b6:172.27.56.9:9858, ee
20b6291-d898-46de-8cb2-861523aed1a3:172.27.87.64:9858,
b7fbd501-27ae-4304-8c42-a612915094c6:172.27.17.133:9858], old=null at 2
2018-10-10 21:50:42,729 WARN impl.LogAppender (LogUtils.java:warn(135)) -
7c1a32b5-34ed-4a2a-aa07-ac75d25858b6: Failed appendEntries to
e20b6291-d898-46de-8cb2-861523aed1a3:172.27.87.64:9858: org.apache..
ratis.shaded.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2018-10-10 21:50:43,245 WARN impl.LogAppender (LogUtils.java:warn(135)) -
7c1a32b5-34ed-4a2a-aa07-ac75d25858b6: Failed appendEntries to
e20b6291-d898-46de-8cb2-861523aed1a3:172.27.87.64:9858: org.apache..
ratis.shaded.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2018-10-10 21:50:43,310 ERROR impl.RaftServerImpl
(RaftServerImpl.java:applyLogToStateMachine(1153)) -
7c1a32b5-34ed-4a2a-aa07-ac75d25858b6: applyTransaction failed for index:1
proto:(t:2, i:1)SMLOGENTRY,,
client-894EC0846FDF, cid=0
2018-10-10 21:50:43,313 ERROR impl.StateMachineUpdater
(ExitUtils.java:terminate(86)) - Terminating with exit status 2:
StateMachineUpdater-7c1a32b5-34ed-4a2a-aa07-ac75d25858b6: the
StateMachineUpdater hii
ts Throwable
java.lang.NoSuchMethodError:
org.apache.hadoop.metrics2.util.MBeans.register(Ljava/lang/String;Ljava/lang/String;Ljava/util/Map;Ljava/lang/Object;)Ljavax/management/ObjectName;
at org.apache.hadoop.utils.RocksDBStore.(RocksDBStore.java:74)
at
org.apache.hadoop.utils.MetadataStoreBuilder.build(MetadataStoreBuilder.java:142)
at
org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil.createContainerMetaData(KeyValueContainerUtil.java:78)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.create(KeyValueContainer.java:133)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handleCreateContainer(KeyValueHandler.java:256)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:179)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:142)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:223)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:229)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.access$300(ContainerStateMachine.java:115)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine$StateMachineHelper.handleCreateContainer(ContainerStateMachine.java:618)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine$StateMachineHelper.executeContainerCommand(ContainerStateMachine.java:642)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.applyTransaction(ContainerStateMachine.java:396)
at
org.apache.ratis.server.impl.RaftServerImpl.applyLogToStateMachine(RaftServerImpl.java:1150)
at
org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:148)
at java.lang.Thread.run(Thread.java:748)
2018-10-10 21:50:43,320 INFO datanode.DataNode (LogAdapter.java:info(51)) -
SHUTDOWN_MSG:
/
SHUTDOWN_MSG: Shutting down DataNode at
ctr-e138-1518143905142-510793-01-02.hwx.site/172.27.56.9
/
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-612) Even after setting hdds.scm.chillmode.enabled to false, SCM allocateblock fails with ChillModePrecheck exception
Namit Maheshwari created HDDS-612:
-
Summary: Even after setting hdds.scm.chillmode.enabled to false,
SCM allocateblock fails with ChillModePrecheck exception
Key: HDDS-612
URL: https://issues.apache.org/jira/browse/HDDS-612
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
{code:java}
2018-10-09 23:11:58,047 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0
on 9863, call Call#70 Retry#0
org.apache.hadoop.ozone.protocol.ScmBlockLocationProtocol.allocateScmBlock from
172.27.56.9:53442
org.apache.hadoop.hdds.scm.exceptions.SCMException: ChillModePrecheck failed
for allocateBlock
at
org.apache.hadoop.hdds.scm.server.ChillModePrecheck.check(ChillModePrecheck.java:38)
at
org.apache.hadoop.hdds.scm.server.ChillModePrecheck.check(ChillModePrecheck.java:30)
at org.apache.hadoop.hdds.scm.ScmUtils.preCheck(ScmUtils.java:42)
at
org.apache.hadoop.hdds.scm.block.BlockManagerImpl.allocateBlock(BlockManagerImpl.java:191)
at
org.apache.hadoop.hdds.scm.server.SCMBlockProtocolServer.allocateBlock(SCMBlockProtocolServer.java:143)
at
org.apache.hadoop.ozone.protocolPB.ScmBlockLocationProtocolServerSideTranslatorPB.allocateScmBlock(ScmBlockLocationProtocolServerSideTranslatorPB.java:74)
at
org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:6255)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-611) SCM UI is not reflecting the changes done in ozone-site.xml
Namit Maheshwari created HDDS-611:
-
Summary: SCM UI is not reflecting the changes done in
ozone-site.xml
Key: HDDS-611
URL: https://issues.apache.org/jira/browse/HDDS-611
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
Attachments: Screen Shot 2018-10-09 at 4.49.58 PM.png
ozone-site.xml was updated to change hdds.scm.chillmode.enabled to false. This
is reflected properly as below:
{code:java}
[root@ctr-e138-1518143905142-510793-01-04 bin]# ./ozone getozoneconf
-confKey hdds.scm.chillmode.enabled
2018-10-09 23:52:12,621 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
false
{code}
But the SCM UI does not reflect this change and it still shows the old value of
true. Please see attached screenshot. !Screen Shot 2018-10-09 at 4.49.58 PM.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-610) On restart of SCM it fails to register DataNodes
Namit Maheshwari created HDDS-610:
-
Summary: On restart of SCM it fails to register DataNodes
Key: HDDS-610
URL: https://issues.apache.org/jira/browse/HDDS-610
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
{code:java}
2018-10-09 23:34:11,105 INFO
org.apache.hadoop.hdds.scm.server.StorageContainerManager: STARTUP_MSG:
/
STARTUP_MSG: Starting StorageContainerManager
STARTUP_MSG: host =
ctr-e138-1518143905142-510793-01-04.hwx.site/172.27.79.197
STARTUP_MSG: args = []
STARTUP_MSG: version = 3.3.0-SNAPSHOT
STARTUP_MSG: classpath =
[jira] [Created] (HDDS-609) Mapreduce example fails with Allocate block failed, error:INTERNAL_ERROR
Namit Maheshwari created HDDS-609:
-
Summary: Mapreduce example fails with Allocate block failed,
error:INTERNAL_ERROR
Key: HDDS-609
URL: https://issues.apache.org/jira/browse/HDDS-609
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
{code:java}
-bash-4.2$ /usr/hdp/current/hadoop-client/bin/hadoop jar
/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar
wordcount /tmp/mr_jobs/input/ o3://bucket2.volume2/mr_job5
18/10/09 23:37:07 INFO conf.Configuration: Removed undeclared tags:
18/10/09 23:37:08 INFO conf.Configuration: Removed undeclared tags:
18/10/09 23:37:08 INFO conf.Configuration: Removed undeclared tags:
18/10/09 23:37:08 INFO client.AHSProxy: Connecting to Application History
server at ctr-e138-1518143905142-510793-01-04.hwx.site/172.27.79.197:10200
18/10/09 23:37:08 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm2
18/10/09 23:37:09 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding
for path: /user/hdfs/.staging/job_1539125785626_0007
18/10/09 23:37:09 INFO input.FileInputFormat: Total input files to process : 1
18/10/09 23:37:09 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
18/10/09 23:37:09 INFO lzo.LzoCodec: Successfully loaded & initialized
native-lzo library [hadoop-lzo rev 5d6248d8d690f8456469979213ab2e9993bfa2e9]
18/10/09 23:37:09 INFO mapreduce.JobSubmitter: number of splits:1
18/10/09 23:37:09 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1539125785626_0007
18/10/09 23:37:09 INFO mapreduce.JobSubmitter: Executing with tokens: []
18/10/09 23:37:10 INFO conf.Configuration: Removed undeclared tags:
18/10/09 23:37:10 INFO conf.Configuration: found resource resource-types.xml at
file:/etc/hadoop/3.0.3.0-63/0/resource-types.xml
18/10/09 23:37:10 INFO conf.Configuration: Removed undeclared tags:
18/10/09 23:37:10 INFO impl.YarnClientImpl: Submitted application
application_1539125785626_0007
18/10/09 23:37:10 INFO mapreduce.Job: The url to track the job:
http://ctr-e138-1518143905142-510793-01-05.hwx.site:8088/proxy/application_1539125785626_0007/
18/10/09 23:37:10 INFO mapreduce.Job: Running job: job_1539125785626_0007
18/10/09 23:37:17 INFO mapreduce.Job: Job job_1539125785626_0007 running in
uber mode : false
18/10/09 23:37:17 INFO mapreduce.Job: map 0% reduce 0%
18/10/09 23:37:24 INFO mapreduce.Job: map 100% reduce 0%
18/10/09 23:37:29 INFO mapreduce.Job: Task Id :
attempt_1539125785626_0007_r_00_0, Status : FAILED
Error: java.io.IOException: Allocate block failed, error:INTERNAL_ERROR
at
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.allocateBlock(OzoneManagerProtocolClientSideTranslatorPB.java:576)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.allocateNewBlock(ChunkGroupOutputStream.java:475)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleWrite(ChunkGroupOutputStream.java:271)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.write(ChunkGroupOutputStream.java:250)
at
org.apache.hadoop.fs.ozone.OzoneFSOutputStream.write(OzoneFSOutputStream.java:47)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:57)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at
org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.writeObject(TextOutputFormat.java:78)
at
org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.write(TextOutputFormat.java:93)
at
org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:559)
at
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at
org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
at org.apache.hadoop.examples.WordCount$IntSumReducer.reduce(WordCount.java:64)
at org.apache.hadoop.examples.WordCount$IntSumReducer.reduce(WordCount.java:52)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:628)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
18/10/09 23:37:35 INFO mapreduce.Job: Task Id :
attempt_1539125785626_0007_r_00_1, Status : FAILED
Error: java.io.IOException: Allocate block failed, error:INTERNAL_ERROR
at
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.allocateBlock(OzoneManagerProtocolClientSideTranslatorPB.java:576)
at
[jira] [Created] (HDDS-608) Mapreduce example fails with Access denied for user hdfs. Superuser privilege is required
Namit Maheshwari created HDDS-608:
-
Summary: Mapreduce example fails with Access denied for user hdfs.
Superuser privilege is required
Key: HDDS-608
URL: https://issues.apache.org/jira/browse/HDDS-608
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
Right now only the administrators can submit a MR job. All the other users
including hdfs will fail with below error:
{code:java}
-bash-4.2$ ./ozone sh bucket create /volume2/bucket2
2018-10-09 23:03:46,399 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2018-10-09 23:03:47,473 INFO rpc.RpcClient: Creating Bucket: volume2/bucket2,
with Versioning false and Storage Type set to DISK
-bash-4.2$ /usr/hdp/current/hadoop-client/bin/hadoop jar
/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar
wordcount /tmp/mr_jobs/input/ o3://bucket2.volume2/mr_job
18/10/09 23:04:08 INFO conf.Configuration: Removed undeclared tags:
18/10/09 23:04:10 INFO conf.Configuration: Removed undeclared tags:
18/10/09 23:04:10 INFO conf.Configuration: Removed undeclared tags:
18/10/09 23:04:10 INFO client.AHSProxy: Connecting to Application History
server at ctr-e138-1518143905142-510793-01-04.hwx.site/172.27.79.197:10200
18/10/09 23:04:10 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm2
18/10/09 23:04:10 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding
for path: /user/hdfs/.staging/job_1539125785626_0003
18/10/09 23:04:11 INFO input.FileInputFormat: Total input files to process : 1
18/10/09 23:04:11 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
18/10/09 23:04:11 INFO lzo.LzoCodec: Successfully loaded & initialized
native-lzo library [hadoop-lzo rev 5d6248d8d690f8456469979213ab2e9993bfa2e9]
18/10/09 23:04:11 INFO mapreduce.JobSubmitter: number of splits:1
18/10/09 23:04:12 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1539125785626_0003
18/10/09 23:04:12 INFO mapreduce.JobSubmitter: Executing with tokens: []
18/10/09 23:04:12 INFO conf.Configuration: Removed undeclared tags:
18/10/09 23:04:12 INFO conf.Configuration: found resource resource-types.xml at
file:/etc/hadoop/3.0.3.0-63/0/resource-types.xml
18/10/09 23:04:12 INFO conf.Configuration: Removed undeclared tags:
18/10/09 23:04:12 INFO impl.YarnClientImpl: Submitted application
application_1539125785626_0003
18/10/09 23:04:12 INFO mapreduce.Job: The url to track the job:
http://ctr-e138-1518143905142-510793-01-05.hwx.site:8088/proxy/application_1539125785626_0003/
18/10/09 23:04:12 INFO mapreduce.Job: Running job: job_1539125785626_0003
18/10/09 23:04:22 INFO mapreduce.Job: Job job_1539125785626_0003 running in
uber mode : false
18/10/09 23:04:22 INFO mapreduce.Job: map 0% reduce 0%
18/10/09 23:04:30 INFO mapreduce.Job: map 100% reduce 0%
18/10/09 23:04:36 INFO mapreduce.Job: Task Id :
attempt_1539125785626_0003_r_00_0, Status : FAILED
Error: org.apache.hadoop.ipc.RemoteException(java.io.IOException): Access
denied for user hdfs. Superuser privilege is required.
at
org.apache.hadoop.hdds.scm.server.StorageContainerManager.checkAdminAccess(StorageContainerManager.java:830)
at
org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer.getContainerWithPipeline(SCMClientProtocolServer.java:190)
at
org.apache.hadoop.ozone.protocolPB.StorageContainerLocationProtocolServerSideTranslatorPB.getContainerWithPipeline(StorageContainerLocationProtocolServerSideTranslatorPB.java:128)
at
org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos$StorageContainerLocationProtocolService$2.callBlockingMethod(StorageContainerLocationProtocolProtos.java:12392)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497)
at org.apache.hadoop.ipc.Client.call(Client.java:1443)
at org.apache.hadoop.ipc.Client.call(Client.java:1353)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy19.getContainerWithPipeline(Unknown Source)
at
org.apache.hadoop.hdds.scm.protocolPB.StorageContainerLocationProtocolClientSideTranslatorPB.getContainerWithPipeline(StorageContainerLocationProtocolClientSideTranslatorPB.java:156)
at
[jira] [Created] (HDDS-600) Mapreduce example fails with java.lang.IllegalArgumentException: Bucket or Volume name has an unsupported character
Namit Maheshwari created HDDS-600:
-
Summary: Mapreduce example fails with
java.lang.IllegalArgumentException: Bucket or Volume name has an unsupported
character
Key: HDDS-600
URL: https://issues.apache.org/jira/browse/HDDS-600
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
Set up a hadoop cluster where ozone is also installed. Ozone can be referenced
via o3://xx.xx.xx.xx:9889
{code:java}
[root@ctr-e138-1518143905142-510793-01-02 ~]# ozone sh bucket list
o3://xx.xx.xx.xx:9889/volume1/
2018-10-09 07:21:24,624 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
[ {
"volumeName" : "volume1",
"bucketName" : "bucket1",
"createdOn" : "Tue, 09 Oct 2018 06:48:02 GMT",
"acls" : [ {
"type" : "USER",
"name" : "root",
"rights" : "READ_WRITE"
}, {
"type" : "GROUP",
"name" : "root",
"rights" : "READ_WRITE"
} ],
"versioning" : "DISABLED",
"storageType" : "DISK"
} ]
[root@ctr-e138-1518143905142-510793-01-02 ~]# ozone sh key list
o3://xx.xx.xx.xx:9889/volume1/bucket1
2018-10-09 07:21:54,500 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
[ {
"version" : 0,
"md5hash" : null,
"createdOn" : "Tue, 09 Oct 2018 06:58:32 GMT",
"modifiedOn" : "Tue, 09 Oct 2018 06:58:32 GMT",
"size" : 0,
"keyName" : "mr_job_dir"
} ]
[root@ctr-e138-1518143905142-510793-01-02 ~]#{code}
Hdfs is also set fine as below
{code:java}
[root@ctr-e138-1518143905142-510793-01-02 ~]# hdfs dfs -ls
/tmp/mr_jobs/input/
Found 1 items
-rw-r--r-- 3 root hdfs 215755 2018-10-09 06:37
/tmp/mr_jobs/input/wordcount_input_1.txt
[root@ctr-e138-1518143905142-510793-01-02 ~]#{code}
Now try to run Mapreduce example job against ozone o3:
{code:java}
[root@ctr-e138-1518143905142-510793-01-02 ~]#
/usr/hdp/current/hadoop-client/bin/hadoop jar
/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar
wordcount /tmp/mr_jobs/input/
o3://xx.xx.xx.xx:9889/volume1/bucket1/mr_job_dir/output
18/10/09 07:15:38 INFO conf.Configuration: Removed undeclared tags:
java.lang.IllegalArgumentException: Bucket or Volume name has an unsupported
character : :
at
org.apache.hadoop.hdds.scm.client.HddsClientUtils.verifyResourceName(HddsClientUtils.java:143)
at
org.apache.hadoop.ozone.client.rpc.RpcClient.getVolumeDetails(RpcClient.java:231)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.ozone.client.OzoneClientInvocationHandler.invoke(OzoneClientInvocationHandler.java:54)
at com.sun.proxy.$Proxy16.getVolumeDetails(Unknown Source)
at org.apache.hadoop.ozone.client.ObjectStore.getVolume(ObjectStore.java:92)
at
org.apache.hadoop.fs.ozone.OzoneFileSystem.initialize(OzoneFileSystem.java:121)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(FileOutputFormat.java:178)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
18/10/09 07:15:39 INFO conf.Configuration: Removed undeclared tags:
[root@ctr-e138-1518143905142-510793-01-02 ~]#
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Created] (HDDS-590) Add unit test for HDDS-583
Namit Maheshwari created HDDS-590: - Summary: Add unit test for HDDS-583 Key: HDDS-590 URL: https://issues.apache.org/jira/browse/HDDS-590 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Namit Maheshwari -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-583) SCM returns zero as the return code, even when invalid options are passed
Namit Maheshwari created HDDS-583:
-
Summary: SCM returns zero as the return code, even when invalid
options are passed
Key: HDDS-583
URL: https://issues.apache.org/jira/browse/HDDS-583
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
While doing testing for HDDS-564, found that SCM returns zero as the return
code, even when invalid options are passed. In StorageContainerManager.java,
please see below code
{code:java}
private static StartupOption parseArguments(String[] args) {
int argsLen = (args == null) ? 0 : args.length;
StartupOption startOpt = StartupOption.HELP;
{code}
Here, startOpt is initialized to HELP, so by default even if wrong options are
passed, parseArguments method returns the value to HELP. This causes the exit
code to be 0.
Ideally, startOpt should be set to null, which will enable it to return non
zero exit code, if the options are invalid.
{code:java}
StartupOption startOpt = null{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-564) Update docker-hadoop-runner branch to reflect changes done in HDDS-490
Namit Maheshwari created HDDS-564: - Summary: Update docker-hadoop-runner branch to reflect changes done in HDDS-490 Key: HDDS-564 URL: https://issues.apache.org/jira/browse/HDDS-564 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Namit Maheshwari starter.sh needs to be modified to reflect the changes done in HDDS-490 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-549) Documentation for key rename is missing in keycommands.md
Namit Maheshwari created HDDS-549: - Summary: Documentation for key rename is missing in keycommands.md Key: HDDS-549 URL: https://issues.apache.org/jira/browse/HDDS-549 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Namit Maheshwari -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-541) ozone volume quota is not honored
Namit Maheshwari created HDDS-541:
-
Summary: ozone volume quota is not honored
Key: HDDS-541
URL: https://issues.apache.org/jira/browse/HDDS-541
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
Create a volume with just 1 MB as quota
{code:java}
[root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone sh volume create
--quota=1MB --user=root /hive
2018-09-23 02:10:11,283 [main] INFO - Creating Volume: hive, with root as owner
and quota set to 1048576 bytes.
{code}
Now create a bucket and put a big key greater than 1MB in the bucket
{code:java}
[root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone sh bucket create
/hive/bucket1
2018-09-23 02:10:38,003 [main] INFO - Creating Bucket: hive/bucket1, with
Versioning false and Storage Type set to DISK
[root@ctr-e138-1518143905142-481027-01-02 bin]# ls -l
../../ozone-0.3.0-SNAPSHOT.tar.gz
-rw-r--r-- 1 root root 165903437 Sep 21 13:16 ../../ozone-0.3.0-SNAPSHOT.tar.gz
[root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone sh key put
/hive/ozone-0.3.0-SNAPSHOT.tar.gz ../../ozone-0.3.0-SNAPSHOT.tar.gz
volume/bucket/key name required in putKey
[root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone sh key put
/hive/bucket1/ozone-0.3.0-SNAPSHOT.tar.gz ../../ozone-0.3.0-SNAPSHOT.tar.gz
[root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone sh key info
/hive/bucket1/ozone-0.3.0-SNAPSHOT.tar.gz
{
"version" : 0,
"md5hash" : null,
"createdOn" : "Sun, 23 Sep 2018 02:13:02 GMT",
"modifiedOn" : "Sun, 23 Sep 2018 02:13:08 GMT",
"size" : 165903437,
"keyName" : "ozone-0.3.0-SNAPSHOT.tar.gz",
"keyLocations" : [ {
"containerID" : 2,
"localID" : 100772661343420416,
"length" : 134217728,
"offset" : 0
}, {
"containerID" : 3,
"localID" : 100772661661007873,
"length" : 31685709,
"offset" : 0
} ]
}{code}
It was able to put a 165 MB file on a volume with just 1MB quota.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-539) ozone datanode ignores the invalid options
Namit Maheshwari created HDDS-539:
-
Summary: ozone datanode ignores the invalid options
Key: HDDS-539
URL: https://issues.apache.org/jira/browse/HDDS-539
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
ozone datanode command starts datanode and ignores the invalid option, apart
from help
{code:java}
[root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone datanode -help
Starts HDDS Datanode
{code}
For all the other invalid options, it just ignores and starts the DN like below:
{code:java}
[root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone datanode -ABC
2018-09-22 00:59:34,462 [main] INFO - STARTUP_MSG:
/
STARTUP_MSG: Starting HddsDatanodeService
STARTUP_MSG: host =
ctr-e138-1518143905142-481027-01-02.hwx.site/172.27.54.20
STARTUP_MSG: args = [-ABC]
STARTUP_MSG: version = 3.2.0-SNAPSHOT
STARTUP_MSG: classpath =
[jira] [Created] (HDDS-538) ozone scmcli is broken
Namit Maheshwari created HDDS-538:
-
Summary: ozone scmcli is broken
Key: HDDS-538
URL: https://issues.apache.org/jira/browse/HDDS-538
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
None of the below commands work for scmcli
{code:java}
[root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone scmcli list
Missing required option '--start='
Usage: ozone scmcli list [-hV] [-c=] -s=
List containers
-c, --count= Maximum number of containers to list
Default: 20
-h, --help Show this help message and exit.
-s, --start= Container id to start the iteration
-V, --version Print version information and exit.
[root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone scmcli list -c=1
Missing required option '--start='
Usage: ozone scmcli list [-hV] [-c=] -s=
List containers
-c, --count= Maximum number of containers to list
Default: 20
-h, --help Show this help message and exit.
-s, --start= Container id to start the iteration
-V, --version Print version information and exit.
[root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone scmcli list -c=1 -s
Missing required parameter for option '--start' ()
Usage: ozone scmcli list [-hV] [-c=] -s=
List containers
-c, --count= Maximum number of containers to list
Default: 20
-h, --help Show this help message and exit.
-s, --start= Container id to start the iteration
-V, --version Print version information and exit.
[root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone scmcli -s
Unknown option: -s
Possible solutions: --scm, --set
[root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone scmcli --start
Unknown option: --start
Usage: ozone scmcli [-hV] [--verbose] [--scm=] [-D=]...
[COMMAND]
Developer tools to handle SCM specific operations.
--scm= The destination scm (host:port)
--verbose More verbose output. Show the stack trace of the errors.
-D, --set=
-h, --help Show this help message and exit.
-V, --version Print version information and exit.
Commands:
list List containers
info Show information about a specific container
delete Delete container
create Create container
close close container
[root@ctr-e138-1518143905142-481027-01-02 bin]#
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-537) ozone sh vol complains for valid length even when the length is valid for 3 characters
Namit Maheshwari created HDDS-537:
-
Summary: ozone sh vol complains for valid length even when the
length is valid for 3 characters
Key: HDDS-537
URL: https://issues.apache.org/jira/browse/HDDS-537
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
{code:java}
[root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone sh vol info abc
Bucket or Volume length is illegal, valid length is 3-63 characters
{code}
Here, the length is already 3 characters still it throws errors, that valid
length is 3-63 characters
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-536) ozone sh throws exception and show on command line for invalid input
Namit Maheshwari created HDDS-536:
-
Summary: ozone sh throws exception and show on command line for
invalid input
Key: HDDS-536
URL: https://issues.apache.org/jira/browse/HDDS-536
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
{code:java}
[root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone sh vol info o3://as
2018-09-22 00:06:03,123 [main] ERROR - Couldn't create protocol class
org.apache.hadoop.ozone.client.rpc.RpcClient exception:
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:291)
at
org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:169)
at
org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:153)
at
org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:109)
at org.apache.hadoop.ozone.web.ozShell.Handler.verifyURI(Handler.java:100)
at
org.apache.hadoop.ozone.web.ozShell.volume.InfoVolumeHandler.call(InfoVolumeHandler.java:49)
at
org.apache.hadoop.ozone.web.ozShell.volume.InfoVolumeHandler.call(InfoVolumeHandler.java:36)
at picocli.CommandLine.execute(CommandLine.java:919)
at picocli.CommandLine.access$700(CommandLine.java:104)
at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
at
picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:77)
Caused by: java.net.UnknownHostException: Invalid host name: local host is:
(unknown); destination host is: "as":9889; java.net.UnknownHostException; For
more details see: http://wiki.apache.org/hadoop/UnknownHost
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:768)
at org.apache.hadoop.ipc.Client$Connection.(Client.java:449)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1552)
at org.apache.hadoop.ipc.Client.call(Client.java:1403)
at org.apache.hadoop.ipc.Client.call(Client.java:1367)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy10.getServiceList(Unknown Source)
at
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceList(OzoneManagerProtocolClientSideTranslatorPB.java:751)
at
org.apache.hadoop.ozone.client.rpc.RpcClient.getScmAddressForClient(RpcClient.java:154)
at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:126)
... 21 more
Caused by: java.net.UnknownHostException
at org.apache.hadoop.ipc.Client$Connection.(Client.java:450)
... 30 more
Invalid host name: local host is: (unknown); destination host is: "as":9889;
java.net.UnknownHostException; For more details see:
http://wiki.apache.org/hadoop/UnknownHost
[root@ctr-e138-1518143905142-481027-01-02 bin]#
{code}
Ideally, it should just throw error like hdfs below:
{code:java}
[hrt_qa@ctr-e138-1518143905142-483670-01-02 hadoopqe]$ hdfs dfs -ls
s3a://namit54/
18/09/22 00:31:53 INFO impl.MetricsConfig: Loaded properties from
hadoop-metrics2.properties
18/09/22 00:31:53 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period
at 10 second(s).
18/09/22 00:31:53 INFO impl.MetricsSystemImpl: s3a-file-system metrics system
started
ls: Bucket namit54 does not exist
[hrt_qa@ctr-e138-1518143905142-483670-01-02 hadoopqe]$
{code}
And not the entire stack trace
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-535) Ozone Manager tries to start on non OM host
Namit Maheshwari created HDDS-535:
-
Summary: Ozone Manager tries to start on non OM host
Key: HDDS-535
URL: https://issues.apache.org/jira/browse/HDDS-535
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
Steps:
# Start up a Ozone multinode cluster
# Set the values for ozone.om.address, ozone.scm.names etc in ozone-site.xml
# Make sure they are pointing to different nodes
# Start all the services on all hosts - OM, SCM, Datanodes
# Now try to run ozone om command on SCM host. It tries to start OM and fails
as below:
{code:java}
[root@ctr-e138-1518143905142-481027-01-04 bin]# ./ozone om
2018-09-21 23:57:13,612 [main] INFO - STARTUP_MSG:
/
STARTUP_MSG: Starting OzoneManager
STARTUP_MSG: host =
ctr-e138-1518143905142-481027-01-04.hwx.site/172.27.28.66
STARTUP_MSG: args = []
STARTUP_MSG: version = 3.2.0-SNAPSHOT
STARTUP_MSG: classpath =
[jira] [Created] (HDDS-534) ozone jmxget does not work
Namit Maheshwari created HDDS-534:
-
Summary: ozone jmxget does not work
Key: HDDS-534
URL: https://issues.apache.org/jira/browse/HDDS-534
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
{code:java}
[root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone jmxget
ERROR: jmxget is not COMMAND nor fully qualified CLASSNAME.
Usage: ozone [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
OPTIONS is none or any of:
--buildpaths attempt to add class files from build tree
--config dir Hadoop config directory
--daemon (start|status|stop) operate on a daemon
--debug turn on shell script debug mode
--help usage information
--hostnames list[,of,host,names] hosts to use in worker mode
--hosts filename list of hosts to use in worker mode
--loglevel level set the log4j level for this command
--workers turn on worker mode
SUBCOMMAND is one of:
Admin Commands:
jmxget get JMX exported values from NameNode or DataNode.
Client Commands:
classpath prints the class path needed to get the hadoop jar and the required
libraries
envvars display computed Hadoop environment variables
freon runs an ozone data generator
fs run a filesystem command on Ozone file system. Equivalent to 'hadoop fs'
genconf generate minimally required ozone configs and output to ozone-site.xml
in specified path
genesis runs a collection of ozone benchmarks to help with tuning.
getozoneconf get ozone config values from configuration
noz ozone debug tool, convert ozone metadata into relational data
scmcli run the CLI of the Storage Container Manager
sh command line interface for object store operations
version print the version
Daemon Commands:
datanode run a HDDS datanode
om Ozone Manager
scm run the Storage Container Manager service
SUBCOMMAND may print help when invoked w/o parameters or with -h.
{code}
As in the logs above jmxget is an option, but does not work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-533) includeFile and excludeFile options does not work for ozone getozoneconf CLI
Namit Maheshwari created HDDS-533:
-
Summary: includeFile and excludeFile options does not work for
ozone getozoneconf CLI
Key: HDDS-533
URL: https://issues.apache.org/jira/browse/HDDS-533
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
{code:java}
ctr-e138-1518143905142-481027-01-04.hwx.site
[root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone getozoneconf
-excludeFile
ozone getconf is utility for getting configuration information from the config
file.
ozone getconf
[-includeFile] gets the include file path that defines the datanodes that can
join the cluster.
[-excludeFile] gets the exclude file path that defines the datanodes that need
to decommissioned.
[-ozonemanagers]gets list of Ozone Manager nodes in the cluster
[-storagecontainermanagers] gets list of ozone storage container manager
nodes in the cluster
[-confKey [key]]gets a specific key from the configuration
[root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone getozoneconf
-includeFile
ozone getconf is utility for getting configuration information from the config
file.
ozone getconf
[-includeFile] gets the include file path that defines the datanodes that can
join the cluster.
[-excludeFile] gets the exclude file path that defines the datanodes that need
to decommissioned.
[-ozonemanagers]gets list of Ozone Manager nodes in the cluster
[-storagecontainermanagers] gets list of ozone storage container manager
nodes in the cluster
[-confKey [key]]gets a specific key from the configuration
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-532) ozone getozoneconf help shows the name as getconf
Namit Maheshwari created HDDS-532:
-
Summary: ozone getozoneconf help shows the name as getconf
Key: HDDS-532
URL: https://issues.apache.org/jira/browse/HDDS-532
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
{code:java}
[root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone getozoneconf -help
ozone getconf is utility for getting configuration information from the config
file.
ozone getconf
[-includeFile] gets the include file path that defines the datanodes that can
join the cluster.
[-excludeFile] gets the exclude file path that defines the datanodes that need
to decommissioned.
[-ozonemanagers]gets list of Ozone Manager nodes in the cluster
[-storagecontainermanagers] gets list of ozone storage container manager
nodes in the cluster
[-confKey [key]]gets a specific key from the configuration
Generic options supported are:
-conf specify an application configuration file
-D define a value for a given property
-fs specify default filesystem URL to use,
overrides 'fs.defaultFS' property from configurations.
-jt specify a ResourceManager
-files specify a comma-separated list of files to be copied to the
map reduce cluster
-libjars specify a comma-separated list of jar files to be included
in the classpath
-archives specify a comma-separated list of archives to be
unarchived on the compute machines
The general command line syntax is:
command [genericOptions] [commandOptions]
{code}
Now the utility name is "getozoneconf", whereas the help shows the as "getconf"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-531) ozone version command prints some information twice
Namit Maheshwari created HDDS-531:
-
Summary: ozone version command prints some information twice
Key: HDDS-531
URL: https://issues.apache.org/jira/browse/HDDS-531
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
{code:java}
[root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone version
//
//
/ /
/ ///
/
/
/ //
/// /
/ ///
/ // /
// // /
///
//
/// //
/ 0.3.0-SNAPSHOT(Arches)
Source code repository g...@github.com:apache/hadoop.git -r
a2752779ac1545f5e0a52fce3cff02a7007e95fb
Compiled by nmaheshwari on 2018-09-21T10:52Z
Compiled with protoc 2.5.0
>From source with checksum 11d8e28a7fa8994c5c73a39cfce5a87
Using HDDS 0.3.0-SNAPSHOT
Source code repository g...@github.com:apache/hadoop.git -r
a2752779ac1545f5e0a52fce3cff02a7007e95fb
Compiled by nmaheshwari on 2018-09-21T10:52Z
Compiled with protoc 2.5.0
>From source with checksum 468f30b1a9935c6bcbf73aabdc7e2aca
{code}
Lines like below are repeated twice:
{code:java}
Source code repository g...@github.com:apache/hadoop.git -r
a2752779ac1545f5e0a52fce3cff02a7007e95fb
Compiled by nmaheshwari on 2018-09-21T10:52Z
Compiled with protoc 2.5.0{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-492) Add more unit tests to ozonefs robot framework
Namit Maheshwari created HDDS-492: - Summary: Add more unit tests to ozonefs robot framework Key: HDDS-492 URL: https://issues.apache.org/jira/browse/HDDS-492 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Namit Maheshwari Currently there are only couple of tests inside ozonefs.robot We should add more unit tests for the same. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-490) Improve om and scm start up options
Namit Maheshwari created HDDS-490:
-
Summary: Improve om and scm start up options
Key: HDDS-490
URL: https://issues.apache.org/jira/browse/HDDS-490
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
I propose the following changes:
# Rename createObjectStore to format
# Change the flag to use --createObjectStore instead of using
-createObjectStore. It is also applicable to other scm and om startup options.
# Fail to format existing object store. If a user runs:
{code:java}
ozone om -createObjectStore{code}
And there is already an object store, it should give a warning message and exit
the process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-486) Update Ozone Documentation to mention that “scm -init” and “om -createObjectStore” should be executed only once
[ https://issues.apache.org/jira/browse/HDDS-486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Maheshwari resolved HDDS-486. --- Resolution: Fixed Duplicate of HDDS-483 > Update Ozone Documentation to mention that “scm -init” and “om > -createObjectStore” should be executed only once > --- > > Key: HDDS-486 > URL: https://issues.apache.org/jira/browse/HDDS-486 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Namit Maheshwari >Assignee: Namit Maheshwari >Priority: Major > > Update Ozone Documentation to mention that “scm -init” and “om > -createObjectStore” should be executed only once. > If it has not been run before otherwise data will be lost. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-485) Update Ozone Java API documentation to fix complete example
[
https://issues.apache.org/jira/browse/HDDS-485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Namit Maheshwari resolved HDDS-485.
---
Resolution: Fixed
Duplicate of HDDS-483
> Update Ozone Java API documentation to fix complete example
> ---
>
> Key: HDDS-485
> URL: https://issues.apache.org/jira/browse/HDDS-485
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Xiaoyu Yao
>Assignee: Namit Maheshwari
>Priority: Major
> Attachments: HDDS-485.001.patch
>
>
> The complete example misses a line to get the ObjectStore from the client:
> {code:java}
> ObjectStore objectStore = ozClient.getObjectStore();{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-484) Update ozone File system documentation to not overwrite HADOOP_CLASSPATH
[ https://issues.apache.org/jira/browse/HDDS-484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Maheshwari resolved HDDS-484. --- Resolution: Duplicate Duplicate of HDDS-483 > Update ozone File system documentation to not overwrite HADOOP_CLASSPATH > > > Key: HDDS-484 > URL: https://issues.apache.org/jira/browse/HDDS-484 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Namit Maheshwari >Priority: Major > Attachments: HDDS-484.001.patch > > > Update ozone File system documentation to not overwrite HADOOP_CLASSPATH -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-486) Update Ozone Documentation to mention that “scm -init” and “om -createObjectStore” should be executed only once
Namit Maheshwari created HDDS-486: - Summary: Update Ozone Documentation to mention that “scm -init” and “om -createObjectStore” should be executed only once Key: HDDS-486 URL: https://issues.apache.org/jira/browse/HDDS-486 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Namit Maheshwari Update Ozone Documentation to mention that “scm -init” and “om -createObjectStore” should be executed only once. If it has not been run before otherwise data will be lost. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-485) Update Ozone Java API documentation to fix complete example
Namit Maheshwari created HDDS-485:
-
Summary: Update Ozone Java API documentation to fix complete
example
Key: HDDS-485
URL: https://issues.apache.org/jira/browse/HDDS-485
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
The complete example misses a line to get the ObjectStore from the client:
{code:java}
ObjectStore objectStore = ozClient.getObjectStore();{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-484) Update ozone File system documentation to not overwrite HADOOP_CLASSPATH
Namit Maheshwari created HDDS-484: - Summary: Update ozone File system documentation to not overwrite HADOOP_CLASSPATH Key: HDDS-484 URL: https://issues.apache.org/jira/browse/HDDS-484 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Namit Maheshwari Update ozone File system documentation to not overwrite HADOOP_CLASSPATH -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-483) Update ozone File system documentation to use 'sh' instead of 'oz'
Namit Maheshwari created HDDS-483: - Summary: Update ozone File system documentation to use 'sh' instead of 'oz' Key: HDDS-483 URL: https://issues.apache.org/jira/browse/HDDS-483 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Namit Maheshwari -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-457) ozone om -help, scm -help commands cant run unless the service is stopped
Namit Maheshwari created HDDS-457:
-
Summary: ozone om -help, scm -help commands cant run unless the
service is stopped
Key: HDDS-457
URL: https://issues.apache.org/jira/browse/HDDS-457
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
{code:java}
➜ ozone-0.3.0-SNAPSHOT bin/ozone om -help
om is running as process 89242. Stop it first.
➜ ozone-0.3.0-SNAPSHOT bin/ozone scm -help
scm is running as process 73361. Stop it first.
{code}
It runs fine once the service is stopped
{code:java}
➜ ozone-0.3.0-SNAPSHOT bin/ozone --daemon stop scm
➜ ozone-0.3.0-SNAPSHOT bin/ozone scm -help
Usage:
ozone scm [genericOptions] [ -init [ -clusterid ] ]
ozone scm [genericOptions] [ -genclusterid ]
ozone scm [ -help ]
Generic options supported are:
-conf specify an application configuration file
-D define a value for a given property
-fs specify default filesystem URL to use,
overrides 'fs.defaultFS' property from configurations.
-jt specify a ResourceManager
-files specify a comma-separated list of files to be copied to the
map reduce cluster
-libjars specify a comma-separated list of jar files to be included
in the classpath
-archives specify a comma-separated list of archives to be
unarchived on the compute machines
The general command line syntax is:
command [genericOptions] [commandOptions]
{code}
Ideally help command should run fine without the service being stopped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-415) bin/ozone om with incorrect argument first logs all the STARTUP_MSG
Namit Maheshwari created HDDS-415:
-
Summary: bin/ozone om with incorrect argument first logs all the
STARTUP_MSG
Key: HDDS-415
URL: https://issues.apache.org/jira/browse/HDDS-415
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
bin/ozone om with incorrect argument first logs all the STARTUP_MSG
{code:java}
➜ ozone-0.2.1-SNAPSHOT bin/ozone om -hgfj
2018-09-07 12:56:12,391 [main] INFO - STARTUP_MSG:
/
STARTUP_MSG: Starting OzoneManager
STARTUP_MSG: host = HW11469.local/10.22.16.67
STARTUP_MSG: args = [-hgfj]
STARTUP_MSG: version = 3.2.0-SNAPSHOT
STARTUP_MSG: classpath =
[jira] [Created] (HDDS-414) sbin/stop-all.sh does not stop Ozone daemons
Namit Maheshwari created HDDS-414:
-
Summary: sbin/stop-all.sh does not stop Ozone daemons
Key: HDDS-414
URL: https://issues.apache.org/jira/browse/HDDS-414
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
sbin/stop-all.sh does not stop Ozone daemons.
Please see below:
{code:java}
➜ ozone-0.2.1-SNAPSHOT jps
8896 Jps
8224 HddsDatanodeService
8162 OzoneManager
7701 StorageContainerManager
➜ ozone-0.2.1-SNAPSHOT pwd
/tmp/ozone-0.2.1-SNAPSHOT
➜ ozone-0.2.1-SNAPSHOT sbin/stop-all.sh
WARNING: Stopping all Apache Hadoop daemons as nmaheshwari in 10 seconds.
WARNING: Use CTRL-C to abort.
Stopping namenodes on [localhost]
localhost: ssh: connect to host localhost port 22: Connection refused
Stopping datanodes
localhost: ssh: connect to host localhost port 22: Connection refused
Stopping secondary namenodes [HW11469.local]
HW11469.local: ssh: connect to host hw11469.local port 22: Connection refused
2018-09-07 12:38:49,044 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
➜ ozone-0.2.1-SNAPSHOT jps
8224 HddsDatanodeService
8162 OzoneManager
7701 StorageContainerManager
9150 Jps
➜ ozone-0.2.1-SNAPSHOT
{code}
The Ozone daemons processes are not stopped even after sbin/stop-all.sh
finished executing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-413) Ozone freon help needs the Scm and OM running
Namit Maheshwari created HDDS-413:
-
Summary: Ozone freon help needs the Scm and OM running
Key: HDDS-413
URL: https://issues.apache.org/jira/browse/HDDS-413
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
Ozone freon help needs the Scm and OM running
{code:java}
./ozone freon --help
2018-09-07 12:23:28,983 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2018-09-07 12:23:30,203 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9862. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-09-07 12:23:31,204 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9862. Already tried 1 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
^C⏎
HW11767 ~/t/o/bin> jps
52445
86095 Jps{code}
If Scm and Om are running, freon help works fine:
{code:java}
HW11767 ~/t/o/bin> /ozone freon --help
2018-09-07 12:30:18,535 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Options supported are:
-numOfThreads number of threads to be launched for the run.
-validateWrites do random validation of data written into
ozone, only subset of data is validated.
-jsonDirdirectory where json is created.
-mode [online | offline]specifies the mode in which Freon should run.
-source specifies the URL of s3 commoncrawl warc file
to be used when the mode is online.
-numOfVolumes specifies number of Volumes to be created in
offline mode
-numOfBuckets specifies number of Buckets to be created per
Volume in offline mode
-numOfKeys specifies number of Keys to be created per
Bucket in offline mode
-keySize specifies the size of Key in bytes to be
created in offline mode
-help prints usage.{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13887) Remove hadoop-ozone-filesystem dependency on hadoop-ozone-integration-test
Namit Maheshwari created HDFS-13887: --- Summary: Remove hadoop-ozone-filesystem dependency on hadoop-ozone-integration-test Key: HDFS-13887 URL: https://issues.apache.org/jira/browse/HDFS-13887 Project: Hadoop HDFS Issue Type: Bug Reporter: Namit Maheshwari hadoop-ozone-filesystem has dependency on hadoop-ozone-integration-test Ideally filesystem modules should not have dependency on test modules. This will also have issues while developing Unit Tests and trying to instantiate OzoneFileSystem object inside hadoop-ozone-integration-test, as that will create a circular dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-290) putKey is failing with KEY_ALLOCATION_ERROR
Namit Maheshwari created HDDS-290:
-
Summary: putKey is failing with KEY_ALLOCATION_ERROR
Key: HDDS-290
URL: https://issues.apache.org/jira/browse/HDDS-290
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Namit Maheshwari
1. List the buckets in Volume /namit
{code}
hadoop@288c0999be17:~$ ozone oz -listBucket /namit
2018-07-24 18:53:26 WARN NativeCodeLoader:60 - Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
[ {
"volumeName" : "namit",
"bucketName" : "abc",
"createdOn" : "Fri, 29 Jul +50529 22:02:39 GMT",
"acls" : [ {
"type" : "USER",
"name" : "hadoop",
"rights" : "READ_WRITE"
}, {
"type" : "GROUP",
"name" : "users",
"rights" : "READ_WRITE"
} ],
"versioning" : "DISABLED",
"storageType" : "DISK"
}, {
"volumeName" : "namit",
"bucketName" : "hjk",
"createdOn" : "Sat, 30 Jul +50529 10:37:24 GMT",
"acls" : [ {
"type" : "USER",
"name" : "hadoop",
"rights" : "READ_WRITE"
}, {
"type" : "GROUP",
"name" : "users",
"rights" : "READ_WRITE"
} ],
"versioning" : "DISABLED",
"storageType" : "DISK"
} ]
{code}
2. Now list the keys in bucket /namit/abc
{code}
hadoop@288c0999be17:~$ ozone oz -listKey /namit/abc
2018-07-24 18:53:56 WARN NativeCodeLoader:60 - Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
[ ]
{code}
3. Now try to put a key to the bucket. It fails as below:
{code}
hadoop@288c0999be17:~$ cat aa
hgfhjljkjhf
hadoop@288c0999be17:~$ ozone oz -putKey /namit/abc/aa -file aa
2018-07-24 18:54:19 WARN NativeCodeLoader:60 - Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Command Failed : Create key failed, error:KEY_ALLOCATION_ERROR
hadoop@288c0999be17:~$
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-289) While creating bucket everything after '/' is ignored without any warning
Namit Maheshwari created HDDS-289:
-
Summary: While creating bucket everything after '/' is ignored
without any warning
Key: HDDS-289
URL: https://issues.apache.org/jira/browse/HDDS-289
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: Ozone Client
Affects Versions: 0.2.1
Reporter: Namit Maheshwari
Please see below example. Here the user issues command to create bucket like
below. Where /namit is the volume.
{code}
hadoop@288c0999be17:~$ ozone oz -createBucket /namit/hjk/fgh
2018-07-24 00:30:52 WARN NativeCodeLoader:60 - Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
2018-07-24 00:30:52 INFO RpcClient:337 - Creating Bucket: namit/hjk, with
Versioning false and Storage Type set to DISK
{code}
As seen above it just ignored '/fgh'
There should be a Warning / Error message instead of just ignoring everything
after a '/'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13715) diskbalancer does not work if one of the blockpools are empty on a Federated cluster
Namit Maheshwari created HDFS-13715: --- Summary: diskbalancer does not work if one of the blockpools are empty on a Federated cluster Key: HDFS-13715 URL: https://issues.apache.org/jira/browse/HDFS-13715 Project: Hadoop HDFS Issue Type: Bug Components: diskbalancer Reporter: Namit Maheshwari Try to run diskbalancer when one of the blockpools are empty on a Federated cluster. diskbalancer process run and completes successfully within seconds. Actual disk balancing does not happen. cc - [~bharatviswa], [~anu] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10964) Add more unit tests for ACLs
Namit Maheshwari created HDFS-10964: --- Summary: Add more unit tests for ACLs Key: HDFS-10964 URL: https://issues.apache.org/jira/browse/HDFS-10964 Project: Hadoop HDFS Issue Type: Sub-task Components: test Reporter: Namit Maheshwari Assignee: Namit Maheshwari This Jira proposes to add more unit tests to validate ACLs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
