Re: [DISCUSS] Mysql-connector-java is GPL licensed; Hadoop can't use it.

[Owen O'Malley]

You are absolutely correct in pointing out that we should update the setup
instructions to include how to install the mysql connector.

With GPL'ed software, the further we can put it off the better, but adding
new modules to contain this code also introduces more burden on both the
project and the users.

Looking through the patch again, it looks like only place references
'com.mysql.*' at all is
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/security/token/SQLConnectionFactory.java
and it isn't clear why that import is necessary. Of course if you configure
the mysql token store, you'll need the mysql connector at runtime. At
LinkedIn, the mysql token store performed 10x better than the zookeeper one.

.. Owen



On Fri, Jun 9, 2023 at 7:19 AM Wei-Chiu Chuang  wrote:

> Thanks Owen,
>
> Yes I am aware of the optional category X. I was under the impression that
> the "optionality" need to be more explicit. For example, by adding
> true tag.
>
> https://medium.com/@danismaz.furkan/difference-between-optional-true-optional-and-scope-provided-scope-7404ec24fb59
>
> In this case, RBF is used by a small set of Hadoop users, moreover an even
> smaller subset of them will choose to use the mysql backend.
>
> (And there need to be instructions for how to acquire the mysql connector
> jar since it's not shipped in the convenience binary)
>
> On Thu, Jun 8, 2023 at 10:55 PM Owen O'Malley 
> wrote:
>
> > We are allowed to use category X software in optional components.
> > Furthermore, the dependency is marked as provided, so it won't be pulled
> > into a transitive closure.
> >
> > You are right that I should have included a comment about that on the
> > original jira.
> >
> > .. Owen
> >
> >
> > On Fri, Jun 9, 2023 at 1:58 AM Wei-Chiu Chuang
> > 
> > wrote:
> >
> > > Hi community,
> > >
> > > While preparing for 3.3.6 RC, I realized the mysql-connector-java
> > > dependency added by HADOOP-18535
> > > <https://issues.apache.org/jira/browse/HADOOP-18535> is GPL licensed.
> > >
> > >
> > > Source:
> > > https://github.com/mysql/mysql-connector-j/blob/release/8.0/LICENSE
> > > See legal discussion at Apache LEGAL-423
> > > <https://issues.apache.org/jira/browse/LEGAL-423>.
> > >
> > > I looked at the original jira and github PR and I don't think the
> license
> > > issue was noticed.
> > >
> > > Is it possible to get rid of the mysql connector dependency? As far as
> I
> > > can tell the dependency is very limited.
> > >
> > > If not, I guess I'll have to revert the commits for now.
> > >
> >
>

Re: [DISCUSS] Mysql-connector-java is GPL licensed; Hadoop can't use it.

[Owen O'Malley]

The relevant url is https://www.apache.org/legal/resolved.html#optional

On Fri, Jun 9, 2023 at 6:54 AM Owen O'Malley  wrote:

> We are allowed to use category X software in optional components.
> Furthermore, the dependency is marked as provided, so it won't be pulled
> into a transitive closure.
>
> You are right that I should have included a comment about that on the
> original jira.
>
> .. Owen
>
>
> On Fri, Jun 9, 2023 at 1:58 AM Wei-Chiu Chuang
>  wrote:
>
>> Hi community,
>>
>> While preparing for 3.3.6 RC, I realized the mysql-connector-java
>> dependency added by HADOOP-18535
>> <https://issues.apache.org/jira/browse/HADOOP-18535> is GPL licensed.
>>
>>
>> Source:
>> https://github.com/mysql/mysql-connector-j/blob/release/8.0/LICENSE
>> See legal discussion at Apache LEGAL-423
>> <https://issues.apache.org/jira/browse/LEGAL-423>.
>>
>> I looked at the original jira and github PR and I don't think the license
>> issue was noticed.
>>
>> Is it possible to get rid of the mysql connector dependency? As far as I
>> can tell the dependency is very limited.
>>
>> If not, I guess I'll have to revert the commits for now.
>>
>

Re: [DISCUSS] Mysql-connector-java is GPL licensed; Hadoop can't use it.

[Owen O'Malley]

We are allowed to use category X software in optional components.
Furthermore, the dependency is marked as provided, so it won't be pulled
into a transitive closure.

You are right that I should have included a comment about that on the
original jira.

.. Owen


On Fri, Jun 9, 2023 at 1:58 AM Wei-Chiu Chuang 
wrote:

> Hi community,
>
> While preparing for 3.3.6 RC, I realized the mysql-connector-java
> dependency added by HADOOP-18535
>  is GPL licensed.
>
>
> Source:
> https://github.com/mysql/mysql-connector-j/blob/release/8.0/LICENSE
> See legal discussion at Apache LEGAL-423
> .
>
> I looked at the original jira and github PR and I don't think the license
> issue was noticed.
>
> Is it possible to get rid of the mysql connector dependency? As far as I
> can tell the dependency is very limited.
>
> If not, I guess I'll have to revert the commits for now.
>

[jira] [Resolved] (HDFS-16917) Add transfer rate quantile metrics for DataNode reads

[Owen O'Malley (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-16917.
--
Fix Version/s: 3.4.0
   Resolution: Fixed

> Add transfer rate quantile metrics for DataNode reads
> -
>
> Key: HDFS-16917
> URL: https://issues.apache.org/jira/browse/HDFS-16917
> Project: Hadoop HDFS
>  Issue Type: Task
>  Components: datanode
>Reporter: Ravindra Dingankar
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Currently we have the following metrics for datanode reads.
> |BytesRead
> BlocksRead
> TotalReadTime|Total number of bytes read from DataNode
> Total number of blocks read from DataNode
> Total number of milliseconds spent on read operation|
> We would like to add a new quantile metric calculating the transfer rate for 
> datanode reads.
> This will give us a distribution across a window of the read transfer rate 
> for each datanode.
> Quantiles for transfer rate per host will help in identifying issues like 
> hotspotting of datasets as well as finding repetitive slow datanodes.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16890) RBF: Add period state refresh to keep router state near active namenode's

[Owen O'Malley (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-16890.
--
Fix Version/s: 3.4.0
   3.3.6
   Resolution: Fixed

> RBF: Add period state refresh to keep router state near active namenode's
> -
>
> Key: HDFS-16890
> URL: https://issues.apache.org/jira/browse/HDFS-16890
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.6
>
>
> When using the ObserverReadProxyProvider, clients can set 
> *dfs.client.failover.observer.auto-msync-period...* to periodically get the 
> Active namenode's state. When using routers without the 
> ObserverReadProxyProvider, this periodic update is lost.
> In a busy cluster, the Router constantly gets updated with the active 
> namenode's state when
>  # There is a write operation.
>  # There is an operation (read/write) from a new clients.
> However, in the scenario when there are no new clients and no write 
> operations, the state kept in the router can lag behind the active's. The 
> router does update its state with responses from the Observer, but the 
> observer may be lagging behind too.
> We should have a periodic refresh in the router to serve a similar role as 
> *dfs.client.failover.observer.auto-msync-period*



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16901) RBF: Routers should propagate the real user in the UGI via the caller context

[Owen O'Malley (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-16901.
--
Fix Version/s: 3.4.0
   3.3.6
   Resolution: Fixed

Thanks, Simba!

> RBF: Routers should propagate the real user in the UGI via the caller context
> -
>
> Key: HDFS-16901
> URL: https://issues.apache.org/jira/browse/HDFS-16901
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.6
>
>
> If the router receives an operation from a proxyUser, it drops the realUser 
> in the UGI and makes the routerUser the realUser for the operation that goes 
> to the namenode.
> In the namenode UGI logs, we'd like the ability to know the original realUser.
> The router should propagate the realUser from the client call as part of the 
> callerContext.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16895) NamenodeHeartbeatService should use credentials of logged in user

[Owen O'Malley (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-16895.
--
Fix Version/s: 3.4.0
   3.3.5
 Assignee: Hector Sandoval Chaverri
   Resolution: Fixed

> NamenodeHeartbeatService should use credentials of logged in user
> -
>
> Key: HDFS-16895
> URL: https://issues.apache.org/jira/browse/HDFS-16895
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: rbf
>Reporter: Hector Sandoval Chaverri
>Assignee: Hector Sandoval Chaverri
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5
>
>
> NamenodeHeartbeatService has been found to log the errors when querying 
> protected Namenode JMX APIs. We have been able to work around this by running 
> kinit with the DFS_ROUTER_KEYTAB_FILE_KEY and 
> DFS_ROUTER_KERBEROS_PRINCIPAL_KEY on the router.
> While investigating a solution, we found that doing the request as part of a  
> UserGroupInformation.getLoginUser.doAs() call doesn't require to kinit before.
> The error logged is:
> {noformat}
> 2022-08-16 21:35:00,265 ERROR 
> org.apache.hadoop.hdfs.server.federation.router.FederationUtil: Cannot parse 
> JMX output for Hadoop:service=NameNode,name=FSNamesystem* from server 
> ltx1-yugiohnn03-ha1.grid.linkedin.com:50070
> org.apache.hadoop.security.authentication.client.AuthenticationException: 
> Error while authenticating with endpoint: 
> http://ltx1-yugiohnn03-ha1.grid.linkedin.com:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem*
>   at sun.reflect.GeneratedConstructorAccessor55.newInstance(Unknown 
> Source)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.wrapExceptionWithMessage(KerberosAuthenticator.java:232)
>   at 
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:219)
>   at 
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:350)
>   at 
> org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:186)
>   at 
> org.apache.hadoop.hdfs.server.federation.router.FederationUtil.getJmx(FederationUtil.java:82)
>   at 
> org.apache.hadoop.hdfs.server.federation.router.NamenodeHeartbeatService.updateJMXParameters(NamenodeHeartbeatService.java:352)
>   at 
> org.apache.hadoop.hdfs.server.federation.router.NamenodeHeartbeatService.getNamenodeStatusReport(NamenodeHeartbeatService.java:295)
>   at 
> org.apache.hadoop.hdfs.server.federation.router.NamenodeHeartbeatService.updateState(NamenodeHeartbeatService.java:218)
>   at 
> org.apache.hadoop.hdfs.server.federation.router.NamenodeHeartbeatService.periodicInvoke(NamenodeHeartbeatService.java:172)
>   at 
> org.apache.hadoop.hdfs.server.federation.router.PeriodicService$1.run(PeriodicService.java:178)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: 
> org.apache.hadoop.security.authentication.client.AuthenticationException: 
> GSSException: No valid credentials provided (Mechanism level: Failed to find 
> any Kerberos tgt)
>   at 
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:360)
>   at 
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:204)
>   ... 15 more
> Caused by: GSSException: No valid credentials provided (Mechanism level: 
> Failed to find any Kerberos tgt)
>   at 
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
>   at 
> sun.security.jgss.krb5.Krb5Me

[jira] [Resolved] (HDFS-16886) Fix documentation for StateStoreRecordOperations#get(Class ..., Query ...)

[Owen O'Malley (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-16886.
--
Fix Version/s: 3.4.0
   3.3.5
   Resolution: Fixed

> Fix documentation for StateStoreRecordOperations#get(Class ..., Query ...)
> --
>
> Key: HDFS-16886
> URL: https://issues.apache.org/jira/browse/HDFS-16886
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Trivial
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5
>
>
> For {*}StateStoreRecordOperations#get(Class ..., Query ...){*}, when multiple 
> records match, the documentation says a null value should be returned and an 
> IOException should be thrown. Both can't happen.
> I believe the intended behavior is that an IOException is thrown. This is the 
> implementation in {*}StateStoreBaseImpl{*}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16877) Namenode doesn't use alignment context in TestObserverWithRouter

[Owen O'Malley (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-16877.
--
Fix Version/s: 3.4.0
 Assignee: Simbarashe Dzinamarira
   Resolution: Fixed

I've committed this. Thanks, Simba!

> Namenode doesn't use alignment context in TestObserverWithRouter
> 
>
> Key: HDFS-16877
> URL: https://issues.apache.org/jira/browse/HDFS-16877
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs, rbf
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> We need to set "{*}dfs.namenode.state.context.enabled{*}" to true for the 
> namenode to send it's stateId in client responses.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16851) RBF: Add a utility to dump the StateStore

[Owen O'Malley (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-16851.
--
Fix Version/s: 3.3.6
   3.4.0
   Resolution: Fixed

> RBF: Add a utility to dump the StateStore
> -
>
> Key: HDFS-16851
> URL: https://issues.apache.org/jira/browse/HDFS-16851
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: rbf
>Reporter: Owen O'Malley
>    Assignee: Owen O'Malley
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.3.6, 3.4.0
>
>
> It would be useful to have a utility to dump the StateStore for RBF.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16847) RBF: StateStore writer should not commit tmp fail if there was an error in writing the file.

[Owen O'Malley (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-16847.
--
Fix Version/s: 3.4.0
   3.3.5
   Resolution: Fixed

I committed this. Thanks, Simba!

> RBF: StateStore writer should not commit tmp fail if there was an error in 
> writing the file.
> 
>
> Key: HDFS-16847
> URL: https://issues.apache.org/jira/browse/HDFS-16847
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs, rbf
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5
>
>
> The file based implementation of the RBF state store has a commit step that 
> moves a temporary file to a permanent location.
> There is a check to see if the write of the temp file was successfully, 
> however, the code to commit doesn't check the success flag.
> This is the relevant code: 
> [https://github.com/apache/hadoop/blob/7d39abd799a5f801a9fd07868a193205ab500bfa/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/driver/impl/StateStoreFileBaseImpl.java#L369]
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[Owen O'Malley (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-16845.
--
Fix Version/s: 3.4.0
   3.3.5
   2.10.3
   Resolution: Fixed

I just committed this. Thanks, Simba!

> Add configuration flag to enable observer reads on routers without using 
> ObserverReadProxyProvider
> --
>
> Key: HDFS-16845
> URL: https://issues.apache.org/jira/browse/HDFS-16845
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5, 2.10.3
>
>
> In order for clients to have routers forward their reads to observers, the 
> clients must use a proxy with an alignment context. This is currently 
> achieved by using the ObserverReadProxyProvider.
> Using ObserverReadProxyProvider allows backward compatible for client 
> configurations.
> However, the ObserverReadProxyProvider forces an msync on initialization 
> which is not required with routers.
> Performing msync calls is more expensive with routers because the router fans 
> out the cal to all namespaces, so we'd like to avoid this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-16856) [RBF] Refactor router admin command to use HDFS AdminHelper class

[Owen O'Malley (Jira)]

Owen O'Malley created HDFS-16856:


 Summary: [RBF] Refactor router admin command to use HDFS 
AdminHelper class
 Key: HDFS-16856
 URL: https://issues.apache.org/jira/browse/HDFS-16856
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: rbf
Reporter: Owen O'Malley
Assignee: Owen O'Malley


Currently, the router admin class is a bit of a mess with a lot of custom 
programming. We should use the infrastructure that was developed in the 
AdminHelper class to standardize the command processing.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-16851) [RBF] Utility to textually dump the StateStore

[Owen O'Malley (Jira)]

Owen O'Malley created HDFS-16851:


 Summary: [RBF] Utility to textually dump the StateStore
 Key: HDFS-16851
 URL: https://issues.apache.org/jira/browse/HDFS-16851
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: rbf
Reporter: Owen O'Malley
Assignee: Owen O'Malley


It would be useful to have a utility to dump the StateStore for RBF.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16844) [RBF] The routers should be resiliant against exceptions from StateStore

[Owen O'Malley (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-16844.
--
Fix Version/s: 3.4.0
   3.3.5
   Resolution: Fixed

> [RBF] The routers should be resiliant against exceptions from StateStore
> 
>
> Key: HDFS-16844
> URL: https://issues.apache.org/jira/browse/HDFS-16844
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: rbf
>Affects Versions: 3.3.4
>    Reporter: Owen O'Malley
>    Assignee: Owen O'Malley
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5
>
>
> Currently, a single exception from the StateStore will cripple a router by 
> clearing the caches before the replacement is loaded. Since the routers have 
> the information in an in-memory cache, it is better to keep running. There is 
> still the timeout that will push the router into safe-mode if it can't load 
> the state store over a longer period of time.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16843) [RBF] The routers should be resiliant against exceptions from StateStore

[Owen O'Malley (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-16843.
--
Resolution: Duplicate

> [RBF] The routers should be resiliant against exceptions from StateStore
> 
>
> Key: HDFS-16843
> URL: https://issues.apache.org/jira/browse/HDFS-16843
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: rbf
>Affects Versions: 3.3.4
>    Reporter: Owen O'Malley
>    Assignee: Owen O'Malley
>Priority: Major
>
> Currently, a single exception from the StateStore will cripple a router by 
> clearing the caches before the replacement is loaded. Since the routers have 
> the information in an in-memory cache, it is better to keep running. There is 
> still the timeout that will push the router into safe-mode if it can't load 
> the state store over a longer period of time.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-16844) [RBF] The routers should be resiliant against exceptions from StateStore

[Owen O'Malley (Jira)]

Owen O'Malley created HDFS-16844:


 Summary: [RBF] The routers should be resiliant against exceptions 
from StateStore
 Key: HDFS-16844
 URL: https://issues.apache.org/jira/browse/HDFS-16844
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: rbf
Affects Versions: 3.3.4
Reporter: Owen O'Malley
Assignee: Owen O'Malley


Currently, a single exception from the StateStore will cripple a router by 
clearing the caches before the replacement is loaded. Since the routers have 
the information in an in-memory cache, it is better to keep running. There is 
still the timeout that will push the router into safe-mode if it can't load the 
state store over a longer period of time.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-16843) [RBF] The routers should be resiliant against exceptions from StateStore

[Owen O'Malley (Jira)]

Owen O'Malley created HDFS-16843:


 Summary: [RBF] The routers should be resiliant against exceptions 
from StateStore
 Key: HDFS-16843
 URL: https://issues.apache.org/jira/browse/HDFS-16843
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: rbf
Affects Versions: 3.3.4
Reporter: Owen O'Malley
Assignee: Owen O'Malley


Currently, a single exception from the StateStore will cripple a router by 
clearing the caches before the replacement is loaded. Since the routers have 
the information in an in-memory cache, it is better to keep running. There is 
still the timeout that will push the router into safe-mode if it can't load the 
state store over a longer period of time.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16836) StandbyCheckpointer can still trigger rollback fs image after RU is finalized

[Owen O'Malley (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-16836.
--
Fix Version/s: 3.4.0
   3.3.5
   Resolution: Fixed

I just committed this. Thanks, Lei!

> StandbyCheckpointer can still trigger rollback fs image after RU is finalized
> -
>
> Key: HDFS-16836
> URL: https://issues.apache.org/jira/browse/HDFS-16836
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Reporter: Lei Yang
>Assignee: Lei Yang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5
>
>
> StandbyCheckpointer trigger rollback fsimage when RU is started.
> When ru is started, a flag (needRollbackImage) was set to true during edit 
> log replay.
> And it only gets reset to false when doCheckpoint() succeeded.
> Think about following scenario:
>  # Start RU, needRollbackImage is set to true.
>  # doCheckpoint() failed.
>  # RU is finalized.
>  # namesystem.getFSImage().hasRollbackFSImage() is always false since 
> rollback image cannot be generated once RU is over.
>  # needRollbackImage was never set to false.
>  # Checkpoints threshold(1m txns) and period(1hr) are not honored.
> {code:java}
> StandbyCheckpointer:
> void doWork() {
>  
>   doCheckpoint();
>   // reset needRollbackCheckpoint to false only when we finish a ckpt
>   // for rollback image
>   if (needRollbackCheckpoint
>   && namesystem.getFSImage().hasRollbackFSImage()) {
> namesystem.setCreatedRollbackImages(true);
> namesystem.setNeedRollbackFsImage(false);
>   }
>   lastCheckpointTime = now;
> } {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-16778) Separate out the logger for which DN is picked by a DFSInputStream

[Owen O'Malley (Jira)]

Owen O'Malley created HDFS-16778:


 Summary: Separate out the logger for which DN is picked by a 
DFSInputStream
 Key: HDFS-16778
 URL: https://issues.apache.org/jira/browse/HDFS-16778
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Owen O'Malley


Currently, there is no way to know which DN a given stream chose without 
turning on debug for all of DFSClient. I'd like the ability to just get that 
logged.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16767) RBF: Support observer node from Router-Based Federation

[Owen O'Malley (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-16767.
--
Fix Version/s: 3.4.0
   3.3.9
   Resolution: Fixed

I just committed this. Thanks, Simba!

> RBF: Support observer node from Router-Based Federation 
> 
>
> Key: HDFS-16767
> URL: https://issues.apache.org/jira/browse/HDFS-16767
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> Enable routers to direct read calls to observer namenodes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

review please?

[Owen O'Malley]

We have a critical issue that has been occasionally crashing our NameNode's
with thread exhaustion. I'd love a review for the patch -
https://github.com/apache/hadoop/pull/4527

Thanks,
   Owen

[jira] [Resolved] (HDFS-16518) KeyProviderCache close cached KeyProvider with Hadoop ShutdownHookManager

[Owen O'Malley (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-16518.
--
Fix Version/s: 3.4.0
   2.10.2
   3.3.3
   Resolution: Fixed

I committed this. Thanks, Lei!

> KeyProviderCache close cached KeyProvider with Hadoop ShutdownHookManager
> -
>
> Key: HDFS-16518
> URL: https://issues.apache.org/jira/browse/HDFS-16518
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 2.10.0
>Reporter: Lei Yang
>Assignee: Lei Yang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 2.10.2, 3.3.3
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> KeyProvider implements Closable interface but some custom implementation of 
> KeyProvider also needs explicit close in KeyProviderCache. An example is to 
> use custom KeyProvider in DFSClient to integrate read encrypted file on HDFS. 
> KeyProvider  currently gets closed in KeyProviderCache only when cache entry 
> is expired or invalidated. In some cases, this is not happening. This seems 
> related to guava cache.
> This patch is to use hadoop JVM shutdownhookManager to globally cleanup cache 
> entries and thus close KeyProvider using cache hook right after filesystem 
> instance gets closed in a deterministic way.
> {code:java}
> Class KeyProviderCache
> ...
>  public KeyProviderCache(long expiryMs) {
>   cache = CacheBuilder.newBuilder()
> .expireAfterAccess(expiryMs, TimeUnit.MILLISECONDS)
> .removalListener(new RemovalListener() {
>   @Override
>   public void onRemoval(
>   @Nonnull RemovalNotification notification) {
> try {
>   assert notification.getValue() != null;
>   notification.getValue().close();
> } catch (Throwable e) {
>   LOG.error(
>   "Error closing KeyProvider with uri ["
>   + notification.getKey() + "]", e);
> }
>   }
> })
> .build(); 
> }{code}
> We could have made a new function KeyProviderCache#close, have each DFSClient 
> call this function and close KeyProvider at the end of each DFSClient#close 
> call but it will expose another problem to potentially close global cache 
> among different DFSClient instances.
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16517) In 2.10 the distance metric is wrong for non-DN machines

[Owen O'Malley (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-16517.
--
Fix Version/s: 2.10.2
   Resolution: Fixed

> In 2.10 the distance metric is wrong for non-DN machines
> 
>
> Key: HDFS-16517
> URL: https://issues.apache.org/jira/browse/HDFS-16517
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.10.1
>Reporter: Owen O'Malley
>    Assignee: Owen O'Malley
>Priority: Major
>  Labels: pull-request-available
> Fix For: 2.10.2
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> In 2.10, the metric for distance between the client and the data node is 
> wrong for machines that aren't running data nodes (ie. 
> getWeightUsingNetworkLocation). The code works correctly in 3.3+. 
> Currently
>  
> ||Client||DataNode||getWeight||getWeightUsingNetworkLocation||
> |/rack1/node1|/rack1/node1|0|0|
> |/rack1/node1|/rack1/node2|2|2|
> |/rack1/node1|/rack2/node2|4|2|
> |/pod1/rack1/node1|/pod1/rack1/node2|2|2|
> |/pod1/rack1/node1|/pod1/rack2/node2|4|2|
> |/pod1/rack1/node1|/pod2/rack2/node2|6|4|
>  
> This bug will destroy data locality on clusters where the clients share racks 
> with DataNodes, but are running on machines that aren't running DataNodes, 
> such as striping federated HDFS clusters across racks.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-16517) In 2.10 the distance metric is wrong for non-DN machines

[Owen O'Malley (Jira)]

Owen O'Malley created HDFS-16517:


 Summary: In 2.10 the distance metric is wrong for non-DN machines
 Key: HDFS-16517
 URL: https://issues.apache.org/jira/browse/HDFS-16517
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 2.10.1
Reporter: Owen O'Malley
Assignee: Owen O'Malley


In 2.10, the metric for distance between the client and the data node is wrong 
for machines that aren't running data nodes (ie. 
getWeightUsingNetworkLocation). The code works correctly in 3.3+.

Currently

 
||Client||DataNode ||getWeight||getWeightUsingNetworkLocation||
|/rack1/node1|/rack1/node1|0|0|
|/rack1/node1|/rack1/node2|2|2|
|/rack1/node1|/rack2/node2|4|2|
|/pod1/rack1/node1|/pod1/rack1/node2|2|2|
|/pod1/rack1/node1|/pod1/rack2/node2|4|2|
|/pod1/rack1/node1|/pod2/rack2/node2|6|4|

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16495) RBF should prepend the client ip rather than append it.

[Owen O'Malley (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-16495.
--
Fix Version/s: 3.4.0
   3.2.4
   Resolution: Fixed

> RBF should prepend the client ip rather than append it.
> ---
>
> Key: HDFS-16495
> URL: https://issues.apache.org/jira/browse/HDFS-16495
> Project: Hadoop HDFS
>  Issue Type: Improvement
>    Reporter: Owen O'Malley
>Assignee: Owen O'Malley
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Currently the Routers append the client ip to the caller context if and only 
> if it is not already set. This would allow the user to fake their ip by 
> setting the caller context. Much better is to prepend it unconditionally.
> The NN must be able to trust the client ip from the caller context.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-16495) RBF should prepend the client ip rather than append it.

[Owen O'Malley (Jira)]

Owen O'Malley created HDFS-16495:


 Summary: RBF should prepend the client ip rather than append it.
 Key: HDFS-16495
 URL: https://issues.apache.org/jira/browse/HDFS-16495
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Owen O'Malley
Assignee: Owen O'Malley


Currently the Routers append the client ip to the caller context if and only if 
it is not already set. This would allow the user to fake their ip by setting 
the caller context. Much better is to prepend it unconditionally.

The NN must be able to trust the client ip from the caller context.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Backport of Router-Based Federation to 2.10.

[Owen O'Malley]

It is because LinkedIn is on Hadoop 2.10 and we want to deploy RBF before we 
move to Hadoop 3. (The big migration for this year is spark 2 to 3.)

.. Owen

> On Dec 11, 2021, at 07:42, Brahma Reddy Battula  wrote:
> 
> Hi Owen,
> 
> out of curiosity , Any reason to continue branc-2.10 ..? As rolling upgrade
> support is available from 2 to 3, any chance to upgrade to branch-3.X,so
> that so many features and improvements will be available..?
> 
> 
>> On Thu, 9 Dec 2021 at 5:55 AM, Owen O'Malley  wrote:
>> 
>> All,
>>   I've started backporting Router-Based Federation work from trunk back
>> into Hadoop 2.10. The goal is to backport everything that is relevant in
>> Hadoop 2. Would anyone outside of LinkedIn be interested in it?
>> 
>> Thanks,
>>Owen
>> 

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Backport of Router-Based Federation to 2.10.

[Owen O'Malley]

All,
   I've started backporting Router-Based Federation work from trunk back
into Hadoop 2.10. The goal is to backport everything that is relevant in
Hadoop 2. Would anyone outside of LinkedIn be interested in it?

Thanks,
Owen

[jira] [Created] (HDFS-16253) Add a toString implementation to DFSInputStream

[Owen O'Malley (Jira)]

Owen O'Malley created HDFS-16253:


 Summary: Add a toString implementation to DFSInputStream
 Key: HDFS-16253
 URL: https://issues.apache.org/jira/browse/HDFS-16253
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Owen O'Malley
Assignee: Owen O'Malley


It would help debugging if there was a useful toString on DFSInputStream.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Collect feedback for HDFS-15638

[Owen O'Malley]

I'm very -1 on adding these semantics.

When you create the table's directory, set the default ACL. That will have
exactly the effect that you are looking for without creating additional
semantics.

.. Owen

On Fri, Oct 16, 2020 at 7:02 PM Xinli shang  wrote:

> Hi all,
>
> I opened https://issues.apache.org/jira/browse/HDFS-15638 and want to
> collect feedback from the community. I know whenever changing the
> permission model that follows POSIX model is never a trivial change. So
> please comment on if you have concerns. For reading convenience, here is a
> copy of the ticket.
>
> *Problem*: Currently, when a user tries to accesses a file he/she needs the
> permissions of it's parent and ancestors and the permission of that file.
> This is correct generally, but for Hive tables directories/files, all the
> files under a partition or even a table usually have the same permissions
> for the same set of ACL groups. Although the permissions and ACL groups are
> the same, the writer still need to call setfacl() for every file to add
> LDAP groups. This results in a huge amount of RPC calls to NN. HDFS has
> default ACL to solve that but that only applies to create and copy, but not
> apply to rename. However, in Hive ETL, rename is very common.
>
> *Proposal*: Add a 1-bit flag to directory inodes to indicate whether or not
> it is a Hive table directory. If that flag is set, then all the
> sub-directory and files under it will just use it's permission and ACL
> groups settings. By doing this way, Hive ETL doesn't need to set
> permissions at the file level. If that flag is not set(by default), work as
> before. To set/unset that flag, it would require admin privilege.
>
> --
> Xinli Shang
>

Re: [DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts

[Owen O'Malley]

I'm very unhappy with this direction. In particular, I don't think git is a
good place for distribution of binary artifacts. Furthermore, the PMC
shouldn't be releasing anything without a release vote.

I'd propose that we make a third party module that contains the *source* of
the pom files to build the relocated jars. This should absolutely be
treated as a last resort for the mostly Google projects that regularly
break binary compatibility (eg. Protobuf & Guava).

In terms of naming, I'd propose something like:

org.apache.hadoop.thirdparty.protobuf2_5
org.apache.hadoop.thirdparty.guava28

In particular, I think we absolutely need to include the version of the
underlying project. On the other hand, since we should not be shading
*everything* we can drop the leading com.google.

The Hadoop project can make releases of  the thirdparty module:


  org.apache.hadoop
  hadoop-thirdparty-protobuf25
  1.0


Note that the version has to be the hadoop thirdparty release number, which
is part of why you need to have the underlying version in the artifact
name. These we can push to maven central as new releases from Hadoop.

Thoughts?

.. Owen

On Fri, Sep 27, 2019 at 8:38 AM Vinayakumar B 
wrote:

> Hi All,
>
>I wanted to discuss about the separate repo for thirdparty dependencies
> which we need to shaded and include in Hadoop component's jars.
>
>Apologies for the big text ahead, but this needs clear explanation!!
>
>Right now most needed such dependency is protobuf. Protobuf dependency
> was not upgraded from 2.5.0 onwards with the fear that downstream builds,
> which depends on transitive dependency protobuf coming from hadoop's jars,
> may fail with the upgrade. Apparently protobuf does not guarantee source
> compatibility, though it guarantees wire compatibility between versions.
> Because of this behavior, version upgrade may cause breakage in known and
> unknown (private?) downstreams.
>
>So to tackle this, we came up the following proposal in HADOOP-13363.
>
>Luckily, As far as I know, no APIs, either public to user or between
> Hadoop processes, is not directly using protobuf classes in signatures. (If
> any exist, please let us know).
>
>Proposal:
>
>
>1. Create a artifact(s) which contains shaded dependencies. All such
> shading/relocation will be with known prefix
> **org.apache.hadoop.thirdparty.**.
>2. Right now protobuf jar (ex: o.a.h.thirdparty:hadoop-shaded-protobuf)
> to start with, all **com.google.protobuf** classes will be relocated as
> **org.apache.hadoop.thirdparty.com.google.protobuf**.
>3. Hadoop modules, which needs protobuf as dependency, will add this
> shaded artifact as dependency (ex:
> o.a.h.thirdparty:hadoop-shaded-protobuf).
>4. All previous usages of "com.google.protobuf" will be relocated to
> "org.apache.hadoop.thirdparty.com.google.protobuf" in the code and will be
> committed. Please note, this replacement is One-Time directly in source
> code, NOT during compile and package.
>5. Once all usages of "com.google.protobuf" is relocated, then hadoop
> dont care about which version of original  "protobuf-java" is in
> dependency.
>6. Just keep "protobuf-java:2.5.0" in dependency tree not to break the
> downstreams. But hadoop will be originally using the latest protobuf
> present in "o.a.h.thirdparty:hadoop-shaded-protobuf".
>
>7. Coming back to separate repo, Following are most appropriate reasons
> of keeping shaded dependency artifact in separate repo instead of
> submodule.
>
>   7a. These artifacts need not be built all the time. It needs to be
> built only when there is a change in the dependency version or the build
> process.
>   7b. If added as "submodule in Hadoop repo", maven-shade-plugin:shade
> will execute only in package phase. That means, "mvn compile" or "mvn
> test-compile" will not be failed as this artifact will not have relocated
> classes, instead it will have original classes, resulting in compilation
> failure. Workaround, build thirdparty submodule first and exclude
> "thirdparty" submodule in other executions. This will be a complex process
> compared to keeping in a separate repo.
>
>   7c. Separate repo, will be a subproject of Hadoop, using the same
> HADOOP jira project, with different versioning prefixed with "thirdparty-"
> (ex: thirdparty-1.0.0).
>   7d. Separate will have same release process as Hadoop.
>
>
> HADOOP-13363 (https://issues.apache.org/jira/browse/HADOOP-13363) is
> an
> umbrella jira tracking the changes to protobuf upgrade.
>
> PR (https://github.com/apache/hadoop-thirdparty/pull/1) has been
> raised
> for separate repo creation in (HADOOP-16595 (
> https://issues.apache.org/jira/browse/HADOOP-16595)
>
> Please provide your inputs for the proposal and review the PR to
> proceed with the proposal.
>
>
>-Thanks,
> Vinay
>
> On Fri, Sep 27, 2019 at 11:54 AM Vinod Kumar Vavilapalli <
> vino...@apache.org>
> wrote:
>
> > Moving the 

Re: [VOTE] Moving Submarine to a separate Apache project proposal

[Owen O'Malley]

Since you don't have any Apache Members, I'll join to provide Apache
oversight.

.. Owen

On Fri, Sep 6, 2019 at 1:38 PM Owen O'Malley  wrote:

> +1 for moving to a new project.
>
> On Sat, Aug 31, 2019 at 10:19 PM Wangda Tan  wrote:
>
>> Hi all,
>>
>> As we discussed in the previous thread [1],
>>
>> I just moved the spin-off proposal to CWIKI and completed all TODO parts.
>>
>>
>> https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Project+Spin-Off+to+TLP+Proposal
>>
>> If you have interests to learn more about this. Please review the proposal
>> let me know if you have any questions/suggestions for the proposal. This
>> will be sent to board post voting passed. (And please note that the
>> previous voting thread [2] to move Submarine to a separate Github repo is
>> a
>> necessary effort to move Submarine to a separate Apache project but not
>> sufficient so I sent two separate voting thread.)
>>
>> Please let me know if I missed anyone in the proposal, and reply if you'd
>> like to be included in the project.
>>
>> This voting runs for 7 days and will be concluded at Sep 7th, 11 PM PDT.
>>
>> Thanks,
>> Wangda Tan
>>
>> [1]
>>
>> https://lists.apache.org/thread.html/4a2210d567cbc05af92c12aa6283fd09b857ce209d537986ed800029@%3Cyarn-dev.hadoop.apache.org%3E
>> [2]
>>
>> https://lists.apache.org/thread.html/6e94469ca105d5a15dc63903a541bd21c7ef70b8bcff475a16b5ed73@%3Cyarn-dev.hadoop.apache.org%3E
>>
>

Re: [VOTE] Moving Submarine to a separate Apache project proposal

[Owen O'Malley]

+1 for moving to a new project.

On Sat, Aug 31, 2019 at 10:19 PM Wangda Tan  wrote:

> Hi all,
>
> As we discussed in the previous thread [1],
>
> I just moved the spin-off proposal to CWIKI and completed all TODO parts.
>
>
> https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Project+Spin-Off+to+TLP+Proposal
>
> If you have interests to learn more about this. Please review the proposal
> let me know if you have any questions/suggestions for the proposal. This
> will be sent to board post voting passed. (And please note that the
> previous voting thread [2] to move Submarine to a separate Github repo is a
> necessary effort to move Submarine to a separate Apache project but not
> sufficient so I sent two separate voting thread.)
>
> Please let me know if I missed anyone in the proposal, and reply if you'd
> like to be included in the project.
>
> This voting runs for 7 days and will be concluded at Sep 7th, 11 PM PDT.
>
> Thanks,
> Wangda Tan
>
> [1]
>
> https://lists.apache.org/thread.html/4a2210d567cbc05af92c12aa6283fd09b857ce209d537986ed800029@%3Cyarn-dev.hadoop.apache.org%3E
> [2]
>
> https://lists.apache.org/thread.html/6e94469ca105d5a15dc63903a541bd21c7ef70b8bcff475a16b5ed73@%3Cyarn-dev.hadoop.apache.org%3E
>

[jira] [Created] (HDFS-14244) hdfs++ doesn't add necessary libraries to dynamic library link

[Owen O'Malley (JIRA)]

Owen O'Malley created HDFS-14244:


 Summary: hdfs++ doesn't add necessary libraries to dynamic library 
link
 Key: HDFS-14244
 URL: https://issues.apache.org/jira/browse/HDFS-14244
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Owen O'Malley
Assignee: Owen O'Malley


When linking with shared libraries, the libhdfs++ cmake file doesn't link 
correctly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[NOTIFICATION] Hadoop trunk rebased

[Owen O'Malley]

As we discussed in hdfs-dev@hadoop, I did a force push to Hadoop's trunk to
replace the Ozone merge with a rebase.

That means that you'll need to rebase your branches.

.. Owen

Re: [RESULT][VOTE] Merge HDDS (HDFS-7240) *code* into trunk

[Owen O'Malley]

Ok, it was bad enough that I spent yesterday afternoon hacking a fix for it.

I basically did a git rebase with always picking the current master version
and then put a new fixup commit to bring the branches into sync.

The original - https://github.com/omalley/hadoop/tree/hdfs-7240
The rebase hack - https://github.com/omalley/hadoop/tree/hdfs-7240-rebase

Compare the git log --graph difference for one of the recent commits:

| * | | | | | | | | | | | | | | | | | | | | | | | commit
919ae746c3a4a424a77e6359af0abe3d7d2c1541
| | | | | | | | | | | | | | | | | | | | | | | | | Author: Weiwei Yang <
w...@apache.org>
| | | | | | | | | | | | | | | | | | | | | | | | | Date:   Wed Nov 1
17:17:03 2017 +0800
| | | | | | | | | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | | | | | | | HDFS-12750. Ozone:
Fix TestStorageContainerManager#testBlockDeletionTransactions. Contributed
by Xiaoyu Yao.
| | | | | | | | | | | | | | | | | | | | | | | | |

versus

* commit 05592658078acef10c3995940510ff09792e
| Author: Weiwei Yang <w...@apache.org>
| Date:   Wed Nov 1 17:17:03 2017 +0800
|
| HDFS-12750. Ozone: Fix
TestStorageContainerManager#testBlockDeletionTransactions. Contributed by
Xiaoyu Yao.
|

I'd like permission to edit the hadoop git history to make the change.

Thoughts?

... Owen

On Wed, Apr 25, 2018 at 11:47 AM, Xiaoyu Yao <x...@hortonworks.com> wrote:

> We followed the process laid out in a mail thread from 2015 that had two
> workflows proposed.
>
> “Git rebase” workflows and “Git merge” workflows.
> Ozone had followed git merge workflow, which merged trunk at a regular
> frequency into ozone, and then ozone was merged back.
>
> Here is the mail that we followed for the merge process.
> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>
> Thanks
> Xiaoyu
>
>
>
>
>
> *From: *Owen O'Malley <owen.omal...@gmail.com>
> *Date: *Wednesday, April 25, 2018 at 11:15 AM
> *To: *Chris Douglas <cdoug...@apache.org>
> *Cc: *Xiaoyu Yao <x...@hortonworks.com>, Jitendra Pandey <
> jiten...@hortonworks.com>, Hdfs-dev <hdfs-dev@hadoop.apache.org>
> *Subject: *Re: [RESULT][VOTE] Merge HDDS (HDFS-7240) *code* into trunk
>
>
>
> *sigh*
>
>
>
> I asked you guys to rebase this before merge. It is tempting to squash it
> into a single commit.
>
>
>
> .. Owen
>
>
>
> On Wed, Apr 25, 2018 at 11:06 AM, Chris Douglas <cdoug...@apache.org>
> wrote:
>
> This really made a mess of trunk. Periodically merging trunk into the
> HDFS-7240 branch, then merging the whole thing back, created a tangle
> of references that's very difficult to work with (look at the output
> of git log --graph).
>
> I'm not sure it's even possible to fix this, but this is why feature
> branches should rebase and create a merge commit with --no-ff. -C
>
>
>
> On Tue, Apr 24, 2018 at 1:20 PM, Xiaoyu Yao <x...@hortonworks.com> wrote:
> > I just merged the branch to trunk (834 commits in total)
> > Again, thanks for all who contributed to HDDS/Ozone!
> >
> > Cheers
> > -Xiaoyu
> >
> > On 4/23/18, 7:26 PM, "Jitendra Pandey" <jiten...@hortonworks.com> wrote:
> >
> > The vote passes with many +1s (12 committers + 5 contributors) and
> no -1.
> >
> > Thanks everyone for voting.
> >
> > On 4/17/18, 5:19 AM, "Jitendra Pandey" <jiten...@hortonworks.com>
> wrote:
> >
> > Hi All,
> >
> >The community unanimously voted (https://s.apache.org/
> HDDSMergeResult) to adopt
> > HDDS/Ozone as a sub-project of Hadoop, here is the formal
> vote for code merge.
> >
> > Here is a quick summary of the code changes:
> >
> > - As decided in the JIRA HDFS-10419, the project has been
> renamed to Hadoop Distributed Data Store or HDDS.
> > - HDDS becomes a sub-project of Hadoop.
> > - Added a Maven profile that disables HDDS compilation by
> default.
> > - The releases of HDDS will be independent of Hadoop and
> will have no impact to current Hadoop release process.
> > - We have made HDDS a loadable module.
> > - Cleaned up changes in HDFS/Hadoop Common to make sure HDDS
> does not impact current users of HDFS.
> >
> > The vote will run for 7 days, I will start this vote with my
> +1.
> >
> > Thanks
> > Jitendra
> >
> >
> > ---
> --
> > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.
> apache.org
> >  

Re: [RESULT][VOTE] Merge HDDS (HDFS-7240) *code* into trunk

[Owen O'Malley]

*sigh*

I asked you guys to rebase this before merge. It is tempting to squash it
into a single commit.

.. Owen

On Wed, Apr 25, 2018 at 11:06 AM, Chris Douglas  wrote:

> This really made a mess of trunk. Periodically merging trunk into the
> HDFS-7240 branch, then merging the whole thing back, created a tangle
> of references that's very difficult to work with (look at the output
> of git log --graph).
>
> I'm not sure it's even possible to fix this, but this is why feature
> branches should rebase and create a merge commit with --no-ff. -C
>
>
> On Tue, Apr 24, 2018 at 1:20 PM, Xiaoyu Yao  wrote:
> > I just merged the branch to trunk (834 commits in total)
> > Again, thanks for all who contributed to HDDS/Ozone!
> >
> > Cheers
> > -Xiaoyu
> >
> > On 4/23/18, 7:26 PM, "Jitendra Pandey" 
> wrote:
> >
> > The vote passes with many +1s (12 committers + 5 contributors) and
> no -1.
> >
> > Thanks everyone for voting.
> >
> > On 4/17/18, 5:19 AM, "Jitendra Pandey" 
> wrote:
> >
> > Hi All,
> >
> >The community unanimously voted (https://s.apache.org/
> HDDSMergeResult) to adopt
> > HDDS/Ozone as a sub-project of Hadoop, here is the formal
> vote for code merge.
> >
> > Here is a quick summary of the code changes:
> >
> > - As decided in the JIRA HDFS-10419, the project has been
> renamed to Hadoop Distributed Data Store or HDDS.
> > - HDDS becomes a sub-project of Hadoop.
> > - Added a Maven profile that disables HDDS compilation by
> default.
> > - The releases of HDDS will be independent of Hadoop and
> will have no impact to current Hadoop release process.
> > - We have made HDDS a loadable module.
> > - Cleaned up changes in HDFS/Hadoop Common to make sure HDDS
> does not impact current users of HDFS.
> >
> > The vote will run for 7 days, I will start this vote with my
> +1.
> >
> > Thanks
> > Jitendra
> >
> >
> > 
> -
> > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.
> apache.org
> > For additional commands, e-mail:
> hdfs-dev-h...@hadoop.apache.org
> >
> >
> >
> >
> >
> >
> > 
> -
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
> >
> >
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

[RESULT][VOTE] Adopt HDSL as a new Hadoop subproject

[Owen O'Malley]

Ok, with a lot of +1's, one +0, and no -1's the vote passes.

We have a new subproject!

We should resolve the final name and then create a jira instance for it.

Thanks everyone,
   Owen

Re: [VOTE] Adopt HDSL as a new Hadoop subproject

[Owen O'Malley]

On Thu, Mar 22, 2018 at 11:09 AM, Lei Xu  wrote:

>
> I have one concrete question about how this HDSL subproject being
> separated: Ozone / HDSL was designed in the current way to re-use the
> existing HDFS code base as much as possible, thus today for this
> container service is in DataNode itself.


Lei,
   I'll let Jitendra and Anu talk to the details of the remaining changes
to HDFS. My understanding is that they are making changes to the patch to
minimize the impact on HDFS. I'm trying to facilitate the proposal for how
to create and manage the subproject.

.. Owen

Re: [VOTE] Adopt HDSL as a new Hadoop subproject

[Owen O'Malley]

On Tue, Mar 20, 2018 at 8:34 PM, 郑锴(铁杰)  wrote:

> >>* HDSL become a subproject of Hadoop.
> I'm not compfortable with the HDSL name, as Konstantin mentioned. H-DSL
> looks like a DSL language at the first glance.
>

Let's start a separate thread about the name. My general guidance on naming
things is to let the people do the work have the biggest say in the naming,
although it is a community decision. It is easy to invest way more time in
the name than it deserves, given that whichever name we end up with it will
come to mean this sub-project. For example, if you google "hdsl hadoop"
you'll already have the right page as the first result.

.. Owen

[VOTE] Adopt HDSL as a new Hadoop subproject

[Owen O'Malley]

All,

Following our discussions on the previous thread (Merging branch HDFS-7240
to trunk), I'd like to propose the following:

* HDSL become a subproject of Hadoop.
* HDSL will release separately from Hadoop. Hadoop releases will not
contain HDSL and vice versa.
* HDSL will get its own jira instance so that the release tags stay
separate.
* On trunk (as opposed to release branches) HDSL will be a separate module
in Hadoop's source tree. This will enable the HDSL to work on their trunk
and the Hadoop trunk without making releases for every change.
* Hadoop's trunk will only build HDSL if a non-default profile is enabled.
* When Hadoop creates a release branch, the RM will delete the HDSL module
from the branch.
* HDSL will have their own Yetus checks and won't cause failures in the
Hadoop patch check.

I think this accomplishes most of the goals of encouraging HDSL development
while minimizing the potential for disruption of HDFS development.

The vote will run the standard 7 days and requires a lazy 2/3 vote. PMC
votes are binding, but everyone is encouraged to vote.

+1 (binding)

.. Owen

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Owen O'Malley]

Andrew and Daryn,
   Do you have any feedback on the proposal? Otherwise, we can start a vote
for "adoption of new codebase" tomorrow.

.. Owen

On Wed, Mar 14, 2018 at 1:50 PM, Owen O'Malley <owen.omal...@gmail.com>
wrote:

> This discussion seems to have died down coming closer consensus without a
> resolution.
>
> I'd like to propose the following compromise:
>
> * HDSL become a subproject of Hadoop.
> * HDSL will release separately from Hadoop. Hadoop releases will not
> contain HDSL and vice versa.
> * HDSL will get its own jira instance so that the release tags stay
> separate.
> * On trunk (as opposed to release branches) HDSL will be a separate module
> in Hadoop's source tree. This will enable the HDSL to work on their trunk
> and the Hadoop trunk without making releases for every change.
> * Hadoop's trunk will only build HDSL if a non-default profile is enabled.
> * When Hadoop creates a release branch, the RM will delete the HDSL module
> from the branch.
> * HDSL will have their own Yetus checks and won't cause failures in the
> Hadoop patch check.
>
> I think this accomplishes most of the goals of encouraging HDSL
> development while minimizing the potential for disruption of HDFS
> development.
>
> Thoughts? Andrew, Jitendra, & Sanjay?
>
> Thanks,
>Owen
>

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Owen O'Malley]

This discussion seems to have died down coming closer consensus without a
resolution.

I'd like to propose the following compromise:

* HDSL become a subproject of Hadoop.
* HDSL will release separately from Hadoop. Hadoop releases will not
contain HDSL and vice versa.
* HDSL will get its own jira instance so that the release tags stay
separate.
* On trunk (as opposed to release branches) HDSL will be a separate module
in Hadoop's source tree. This will enable the HDSL to work on their trunk
and the Hadoop trunk without making releases for every change.
* Hadoop's trunk will only build HDSL if a non-default profile is enabled.
* When Hadoop creates a release branch, the RM will delete the HDSL module
from the branch.
* HDSL will have their own Yetus checks and won't cause failures in the
Hadoop patch check.

I think this accomplishes most of the goals of encouraging HDSL development
while minimizing the potential for disruption of HDFS development.

Thoughts? Andrew, Jitendra, & Sanjay?

Thanks,
   Owen

Re: [VOTE] Merging branch HDFS-8707 (native HDFS client) to trunk

[Owen O'Malley]

+1 (binding)

.. Owen

On Sun, Mar 11, 2018 at 6:20 PM, Chris Douglas  wrote:

> +1 (binding) -C
>
> On Thu, Mar 8, 2018 at 9:31 AM, Jim Clampffer 
> wrote:
> > Hi Everyone,
> >
> > The feedback was generally positive on the discussion thread [1] so I'd
> > like to start a formal vote for merging HDFS-8707 (libhdfs++) into trunk.
> > The vote will be open for 7 days and end 6PM EST on 3/15/18.
> >
> > This branch includes a C++ implementation of an HDFS client for use in
> > applications that don't run an in-process JVM.  Right now the branch only
> > supports reads and metadata calls.
> >
> > Features (paraphrasing the list from the discussion thread):
> > -Avoiding the JVM means applications that use libhdfs++ can explicitly
> > control resources (memory, FDs, threads).  The driving goal for this
> > project was to let C/C++ applications access HDFS while maintaining a
> > single heap.
> > -Includes support for Kerberos authentication.
> > -Includes a libhdfs/libhdfs3 compatible C API as well as a C++ API that
> > supports asynchronous operations.  Applications that only do reads may be
> > able to use this as a drop in replacement for libhdfs.
> > -Asynchronous IO is built on top of boost::asio which in turn uses
> > select/epoll so many sockets can be monitored from a single thread (or
> > thread pool) rather than spawning a thread to sleep on a blocked socket.
> > -Includes a set of utilities written in C++ that mirror the CLI tools
> (e.g.
> > ./hdfs dfs -ls).  These have a 3 order of magnitude lower startup time
> than
> > java client which is useful for scripts that need to work with many
> files.
> > -Support for cancelable reads that release associated resources
> > immediately.  Useful for applications that need to be responsive to
> > interactive users.
> >
> > Other points:
> > -This is almost all new code in a new subdirectory.  No Java source for
> the
> > rest of hadoop was changed so there's no risk of regressions there.  The
> > only changes outside of that subdirectory were integrating the build in
> > some of the pom files and adding a couple dependencies to the DockerFile.
> > -The library has had plenty of burn-in time.  It's been used in
> production
> > for well over a year and is indirectly being distributed as part of the
> > Apache ORC project (in the form of a third party dependency).
> > -There isn't much in the way of well formatted documentation right now.
> > The documentation for the libhdfs API is applicable to the libhdfs++ C
> API.
> > Header files describe various component including details about threading
> > and lifecycle expectations for important objects.  Good places to start
> are
> > hdfspp.h, filesystem.h, filehandle.h, rpc_connection.h and rpc_enginel.h.
> >
> > I'll start with my +1 (binding).
> >
> > [1]
> > http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/
> 201803.mbox/browser
> > (second message in thread, can't figure out how to link directly to mine)
> >
> > Thanks!
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Owen O'Malley]

Hi Joep,

On Tue, Mar 6, 2018 at 6:50 PM, J. Rottinghuis 
wrote:

Obviously when people do want to use Ozone, then having it in the same repo
> is easier. The flipside is that, separate top-level project in the same
> repo or not, it adds to the Hadoop releases.
>

Apache projects are about the group of people who are working together.
There is a large overlap between the team working on HDFS and Ozone, which
is a lot of the motivation to keep project overhead to a minimum and not
start a new project.

Using the same releases or separate releases is a distinct choice. Many
Apache projects, such as Common and Maven, have multiple artifacts that
release independently. In Hive, we have two sub-projects that release
indepdendently: Hive Storage API, and Hive.

One thing we did during that split to minimize the challenges to the
developers was that Storage API and Hive have the same master branch.
However, since they have different releases, they have their own release
branches and release numbers.

If there is a change in Ozone and a new release needed, it would have to
> wait for a Hadoop release. Ditto if there is a Hadoop release and there is
> an issue with Ozone. The case that one could turn off Ozone through a Maven
> profile works only to some extend.
> If we have done a 3.x release with Ozone in it, would it make sense to do
> a 3.y release with y>x without Ozone in it? That would be weird.
>

Actually, if Ozone is marked as unstable/evolving (we should actually have
an even stronger warning for a feature preview), we could remove it in a
3.x. If a user picks up a feature before it is stable, we try to provide a
stable platform, but mistakes happen. Introducing an incompatible change to
the Ozone API between 3.1 and 3.2 wouldn't be good, but it wouldn't be the
end of the world.

.. Owen

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Owen O'Malley]

On Thu, Mar 1, 2018 at 11:03 PM, Andrew Wang 
wrote:

Owen mentioned making a Hadoop subproject; we'd have to
> hash out what exactly this means (I assume a separate repo still managed by
> the Hadoop project), but I think we could make this work if it's more
> attractive than incubation or a new TLP.


Ok, there are multiple levels of sub-projects that all make sense:

   - Same source tree, same releases - examples like HDFS & YARN
   - Same master branch, separate releases and release branches - Hive's
   Storage API vs Hive. It is in the source tree for the master branch, but
   has distinct releases and release branches.
   - Separate source, separate release - Apache Commons.

There are advantages and disadvantages to each. I'd propose that we use the
same source, same release pattern for Ozone. Note that we tried and later
reverted doing Common, HDFS, and YARN as separate source, separate release
because it was too much trouble. I like Daryn's idea of putting it as a top
level directory in Hadoop and making sure that nothing in Common, HDFS, or
YARN depend on it. That way if a Release Manager doesn't think it is ready
for release, it can be trivially removed before the release.

One thing about using the same releases, Sanjay and Jitendra are signing up
to make much more regular bugfix and minor releases in the near future. For
example, they'll need to make 3.2 relatively soon to get it released and
then 3.3 somewhere in the next 3 to 6 months. That would be good for the
project. Hadoop needs more regular releases and fewer big bang releases.

.. Owen

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Owen O'Malley]

I think it would be good to get this in sooner rather than later, but I
have some thoughts.

   1. It is hard to tell what has changed. git rebase -i tells me the
   branch has 722 commits. The rebase failed with a conflict. It would really
   help if you rebased to current trunk.
   2. I think Ozone would be a good Hadoop subproject, but it should be
   outside of HDFS.
   3. CBlock, which is also coming in this merge, would benefit from more
   separation from HDFS.
   4. What are the new transitive dependencies that Ozone, HDSL, and CBlock
   adding to the clients? The servers matter too, but the client dependencies
   have a huge impact on our users.
   5. Have you checked the new dependencies for compatibility with ASL?


On Thu, Mar 1, 2018 at 2:45 PM, Clay B.  wrote:

> Oops, retrying now subscribed to more than solely yarn-dev.
>
> -Clay
>
>
> On Wed, 28 Feb 2018, Clay B. wrote:
>
> +1 (non-binding)
>>
>> I have walked through the code and find it very compelling as a user; I
>> really look forward to seeing the Ozone code mature and it maturing HDFS
>> features together. The points which excite me as an eight year HDFS user
>> are:
>>
>> * Excitement for making the datanode a storage technology container - this
>>  patch clearly brings fresh thought to HDFS keeping it from growing stale
>>
>> * Ability to build upon a shared storage infrastructure for diverse
>>  loads: I do not want to have "stranded" storage capacity or have to
>>  manage competing storage systems on the same disks (and further I want
>>  the metrics datanodes can provide me today, so I do not have to
>>  instrument two systems or evolve their instrumentation separately).
>>
>> * Looking forward to supporting object-sized files!
>>
>> * Moves HDFS in the right direction to test out new block management
>>  techniques for scaling HDFS. I am really excited to see the raft
>>  integration; I hope it opens a new era in Hadoop matching modern systems
>>  design with new consistency and replication options in our ever
>>  distributed ecosystem.
>>
>> -Clay
>>
>> On Mon, 26 Feb 2018, Jitendra Pandey wrote:
>>
>>Dear folks,
>>>   We would like to start a vote to merge HDFS-7240 branch into
>>> trunk. The context can be reviewed in the DISCUSSION thread, and in the
>>> jiras (See references below).
>>>
>>>HDFS-7240 introduces Hadoop Distributed Storage Layer (HDSL), which
>>> is a distributed, replicated block layer.
>>>The old HDFS namespace and NN can be connected to this new block
>>> layer as we have described in HDFS-10419.
>>>We also introduce a key-value namespace called Ozone built on HDSL.
>>>
>>>The code is in a separate module and is turned off by default. In a
>>> secure setup, HDSL and Ozone daemons cannot be started.
>>>
>>>The detailed documentation is available at
>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+
>>> Distributed+Storage+Layer+and+Applications
>>>
>>>
>>>I will start with my vote.
>>>+1 (binding)
>>>
>>>
>>>Discussion Thread:
>>>  https://s.apache.org/7240-merge
>>>  https://s.apache.org/4sfU
>>>
>>>Jiras:
>>>   https://issues.apache.org/jira/browse/HDFS-7240
>>>   https://issues.apache.org/jira/browse/HDFS-10419
>>>   https://issues.apache.org/jira/browse/HDFS-13074
>>>   https://issues.apache.org/jira/browse/HDFS-13180
>>>
>>>
>>>Thanks
>>>jitendra
>>>
>>>
>>>
>>>
>>>
>>>DISCUSSION THREAD SUMMARY :
>>>
>>>On 2/13/18, 6:28 PM, "sanjay Radia" 
>>> wrote:
>>>
>>>Sorry the formatting got messed by my email client.  Here
>>> it is again
>>>
>>>
>>>Dear
>>> Hadoop Community Members,
>>>
>>>   We had multiple community discussions, a few meetings
>>> in smaller groups and also jira discussions with respect to this thread. We
>>> express our gratitude for participation and valuable comments.
>>>
>>>The key questions raised were following
>>>1) How the new block storage layer and OzoneFS benefit
>>> HDFS and we were asked to chalk out a roadmap towards the goal of a
>>> scalable namenode working with the new storage layer
>>>2) We were asked to provide a security design
>>>3)There were questions around stability given ozone
>>> brings in a large body of code.
>>>4) Why can?t they be separate projects forever or merged
>>> in when production ready?
>>>
>>>We have responded to all the above questions with
>>> detailed explanations and answers on the jira as well as in the
>>> discussions. We believe that should sufficiently address community?s
>>> concerns.
>>>
>>>Please see the summary below:
>>>
>>>1) The new code base benefits HDFS scaling and a roadmap
>>> has been provided.
>>>
>>>

Re: [DISCUSS] Merging HDFS-8707 (C++ HDFS client) to trunk

[Owen O'Malley]

+1 on the merge. We've been using it on the trunk of ORC for a while. It
will be great to have it released by Hadoop.

.. Owen

On Thu, Mar 1, 2018 at 10:31 AM, Vinayakumar B 
wrote:

> Definitely this would be great addition. Kudos to everyone's contributions.
>
> I am not a C++ expert. So cannot vote on code.
>
>   ---A libhdfs/libhdfs3 compatible C API that allows libhdfs++ to serve as
> a drop-in replacement for clients that only need read support (until
> libhdfs++
> also supports writes).
>
> Wouldn't it be nice to have write support as well before merge...?
> If everyone feels its okay to have read alone for now, I am okay anyway.
>
> On 1 Mar 2018 11:35 pm, "Jim Clampffer"  wrote:
>
> > Thanks for the feedback Chris and Kai!
> >
> > Chris, do you mean potentially landing this in its current state and
> > handling some of the rough edges after?  I could see this working just
> > because there's no impact on any existing code.
> >
> > With regards to your questions Kai:
> > There isn't a good doc for the internal architecture yet; I just
> reassigned
> > HDFS-9115 to myself to handle that.  Are there any specific areas you'd
> > like to know about so I can prioritize those?
> > Here's some header files that include a lot of comments that should help
> > out for now:
> > -hdfspp.h - main header for the C++ API
> > -filesystem.h and filehandle.h - describes some rules about object
> > lifetimes and threading from the API point of view (most classes have
> > comments describing any restrictions on threading, locking, and
> lifecycle).
> > -rpc_engine.h and rpc_connection.h begin getting into the async RPC
> > implementation.
> >
> >
> > 1) Yes, it's a reimplementation of the entire client in C++.  Using
> > libhdfs3 as a reference helps a lot here but it's still a lot of work.
> > 2) EC isn't supported now, though that'd be great to have, and I agree
> that
> > it's going to be take a lot of effort to implement.  Right now if you
> tried
> > to read an EC file I think you'd get some unhelpful error out of the
> block
> > reader but I don't have an EC enabled cluster set up to test.  Adding an
> > explicit not supported message would be straightforward.
> > 3) libhdfs++ reuses all of the minidfscluster tests that libhdfs already
> > had so we get consistency checks on the C API.  There's a few new tests
> > that also get run on both libhdfs and libhdfs++ and make sure the
> expected
> > output is the same too.
> > 4) I agree, I just haven't had a chance to look into the distribution
> build
> > to see how to do it.  HDFS-9465 is tracking this.
> > 5) Not yet (HDFS-8765).
> >
> > Regards,
> > James
> >
> >
> >
> >
> > On Thu, Mar 1, 2018 at 4:28 AM, 郑锴(铁杰) 
> > wrote:
> >
> > > The work sounds solid and great! + to have this.
> > >
> > > Is there any quick doc to take a glance at? Some quick questions to be
> > > familiar with:
> > > 1. Seems the client is all implemented in c++ without any Java codes
> (so
> > > no JVM overhead), which means lots of work, rewriting HDFS client.
> Right?
> > > 2.  Guess erasure coding feature isn't supported, as it'd involve
> > > significant development, right? If yes, what will it say when read
> > erasure
> > > coded file?
> > > 3. Is there any building/testing mechanism to enforce the consistency
> > > between the c++ part and Java part?
> > > 4. I thought the public header and lib should be exported when building
> > > the distribution package, otherwise hard to use the new C api.
> > > 5. Is the short-circuit read supported?
> > >
> > > Thanks.
> > >
> > >
> > > Regards,
> > > Kai
> > >
> > > --
> > > 发件人：Chris Douglas 
> > > 发送时间：2018年3月1日(星期四) 05:08
> > > 收件人：Jim Clampffer 
> > > 抄 送：Hdfs-dev 
> > > 主 题：Re: [DISCUSS] Merging HDFS-8707 (C++ HDFS client) to trunk
> > >
> > > +1
> > >
> > > Let's get this done. We've had many false starts on a native HDFS
> > > client. This is a good base to build on. -C
> > >
> > > On Wed, Feb 28, 2018 at 9:55 AM, Jim Clampffer
> > >  wrote:
> > > > Hi everyone,
> > > >
> > > > I'd like to start a thread to discuss merging the HDFS-
> > > 8707 aka libhdfs++
> > > > into trunk.  I sent originally sent a similar
> > > email out last October but it
> > > > sounds like it was buried by discussions about other feature merges
> > that
> > > > were going on at the time.
> > > >
> > > > libhdfs++ is an HDFS client written in C++ designed to be used in
> > > > applications that are written in non-JVM based
> > > languages.  In its current
> > > > state it supports kerberos authenticated reads from HDFS
> > > and has been used
> > > > in production clusters for over a year so it has had a
> > > significant amount
> > > > of burn-in time.  The HDFS-8707 branch has been around for about 2
> > years
> 

[CFP] Dataworks Summit San Jose Call for Presentations closes Friday 9 Feb

[Owen O'Malley]

All,

Dataworks Summit San Jose 2018 is June 17-21.   The call for abstracts is
open through February 9th.  One of the tracks is Big Compute And Storage,
which is great for talks about Hadoop.  You can submit an abstract at
https://dataworkssummit.com/san-jose-2018/

Thanks,
   Owen

Hadoop Summit EU 2017

[Owen O'Malley]

The DataWorks Summit EU 2017 (including Hadoop Summit) is going to be in
Munich April 5-6 2017
.  I’ve pasted the text from the CFP below.

Would you like to share your knowledge with the best and brightest in the
data community? If so, we encourage you to submit an abstract for DataWorks
Summit with Hadoop Summit being held on April 5-6, 2017 at The ICM –
International Congress Center Munich.

DataWorks Summit with Hadoop Summit is the premier event for business and
technical audiences who want to learn how data is transforming business and
the underlying technologies that are driving that change.

Our 2017 tracks include:
· Applications
· Enterprise Adoption
· Data Processing & Warehousing
· Apache Hadoop Core Internals
· Governance & Security
· IoT & Streaming
· Cloud & Operations
· Apache Spark & Data Science
For questions or additional information, please contact Joshua Woodward.

Deadline: Friday, November 11, 2017.
Submission Link: http://dataworkssummit.com/munich-2017/abstracts/submit-
abstract/

.. Owen

Re: Hadoop encryption module as Apache Chimera incubator project

[Owen O'Malley]

On Wed, Jan 27, 2016 at 9:59 AM, Gangumalla, Uma 
wrote:

> I think Chimera goal is to enhance even for other use cases.


Naturally.


> For Hadoop, CTR mode should be enough today,


This isn't true. Hadoop should use better encryption for RPC and shuffle,
both of which should not use CTR.


> I think separate module and
> independent release is good idea but I am not so strong on the point to
> keep under Hadoop.


I believe encryption is becoming a core part of Hadoop. I think that moving
core components out of Hadoop is bad from a project management perspective.
To put it another way, a bug in the encryption routines will likely become
a security problem that security@hadoop needs to hear about. I don't think
adding a separate project in the middle of that communication chain is a
good idea. The same applies to data corruption problems, and so on...


> It may be good to keep at generalized place(As in the
> discussion, we thought that place could be Apache Commons).


Apache Commons is a collection of *Java* projects, so Chimera as a
JNI-based library isn't a natural fit. Furthermore, Apache Commons doesn't
have its own security list so problems will go to the generic
secur...@apache.org.

Why do you think that Apache Commons is a better home than Hadoop?

.. Owen

Re: Hadoop encryption module as Apache Chimera incubator project

[Owen O'Malley]

Sorry to be coming in to this discussion late. Rather than pull the code
out of Hadoop, may I suggest instead making it a separate subproject within
Hadoop itself? I'd suggest letting it release independently of Hadoop,
since it will need a much faster cadence that Hadoop proper does. It should
also keep the number of dependencies to a very small set (empty?).

Thoughts?
   Owen

[jira] [Created] (HDFS-9025) fix compilation issues on arch linux

[Owen O'Malley (JIRA)]

Owen O'Malley created HDFS-9025:
---

 Summary: fix compilation issues on arch linux
 Key: HDFS-9025
 URL: https://issues.apache.org/jira/browse/HDFS-9025
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Owen O'Malley


There are several compilation issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8707) Implement an async pure c++ HDFS client

[Owen O'Malley (JIRA)]

Owen O'Malley created HDFS-8707:
---

 Summary: Implement an async pure c++ HDFS client
 Key: HDFS-8707
 URL: https://issues.apache.org/jira/browse/HDFS-8707
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: hdfs-client
Reporter: Owen O'Malley
Assignee: Haohui Mai


As part of working on the C++ ORC reader at ORC-3, we need an HDFS pure C++ 
client that lets us do async io to HDFS. We want to start from the code that 
Haohui's been working on at https://github.com/haohui/libhdfspp .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[DISCUSS] Create branch for HDFS-8707

[Owen O'Malley]

All,
   Haohui and I would like to pull code for an async io c++ hdfs client
from https://github.com/haohui/libhdfspp as a new development branch. The
motivation is to get a high performance asynchronous c++ hdfs client and
the driving need is to support reading ORC files from HDFS without a jvm in
the process. The read path is already working and I think it is a good time
to move this project to Apache.

Thoughts,
   Owen

Re: Moving to JDK7, JDK8 and new major releases

[Owen O'Malley]

On Tue, Jun 24, 2014 at 4:44 PM, Alejandro Abdelnur t...@cloudera.com
wrote:

 After reading this thread and thinking a bit about it, I think it should be
 OK such move up to JDK7 in Hadoop


I agree with Alejandro. Changing minimum JDKs is not an incompatible change
and is fine in the 2 branch. (Although I think it is would *not* be
appropriate for a patch release.) Of course we need to do it with
forethought and testing, but moving off of JDK 6, which is EOL'ed is a good
thing. Moving to Java 8 as a minimum seems much too aggressive and I would
push back on that.

I'm also think that we need to let the dust settle on the Hadoop 2 line for
a while before we talk about Hadoop 3. It seems that it has only been in
the last 6 months that Hadoop 2 adoption has reached the main stream users.
Our user community needs time to digest the changes in Hadoop 2.x before we
fracture the community by starting to discuss Hadoop 3 releases.

.. Owen

[jira] [Resolved] (HDFS-3983) Hftp should support both SPNEGO and KSSL

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-3983.
-

  Resolution: Won't Fix
Target Version/s:   (was: )

KSSL is deprecated and should never be used for secure deployments.

 Hftp should support both SPNEGO and KSSL
 

 Key: HDFS-3983
 URL: https://issues.apache.org/jira/browse/HDFS-3983
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: security
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Assignee: Eli Collins
Priority: Blocker
 Attachments: hdfs-3983.txt, hdfs-3983.txt


 Hftp currently doesn't work against a secure cluster unless you configure 
 {{dfs.https.port}} to be the http port, otherwise the client can't fetch 
 tokens:
 {noformat}
 $ hadoop fs -ls hftp://c1225.hal.cloudera.com:50070/
 12/09/26 18:02:00 INFO fs.FileSystem: Couldn't get a delegation token from 
 http://c1225.hal.cloudera.com:50470 using http.
 ls: Security enabled but user not authenticated by filter
 {noformat}
 This is due to Hftp still using the https port. Post HDFS-2617 it should use 
 the regular http port. Hsftp should still use the secure port, however now 
 that we have HADOOP-8581 it's worth considering removing Hsftp entirely. I'll 
 start a separate thread about that.  



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Resolved] (HDFS-3699) HftpFileSystem should try both KSSL and SPNEGO when authentication is required

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-3699.
-

Resolution: Won't Fix

Using KSSL is strongly deprecated and should be avoided in secure clusters.

 HftpFileSystem should try both KSSL and SPNEGO when authentication is required
 --

 Key: HDFS-3699
 URL: https://issues.apache.org/jira/browse/HDFS-3699
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: eric baldeschwieler

 See discussion in HDFS-2617 (Replaced Kerberized SSL for image transfer and 
 fsck with SPNEGO-based solution).
 To handle the transition from Hadoop1.0 systems running KSSL authentication 
 to Hadoop systems running SPNEGO, it would be good to fix the client in both 
 1 and 2 to try SPNEGO and then fall back to try KSSL.  
 This will allow organizations that are running a lot of Hadoop 1.0 to 
 gradually transition over, without needing to convert all clusters at the 
 same time.  They would first need to update their 1.0 HFTP clients (and 
 2.0/0.23 if they are already running those) and then they could copy data 
 between clusters without needing to move all clusters to SPNEGO in a big bang.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Release numbering for branch-2 releases

[Owen O'Malley]

I think that using -(alpha,beta) tags on the release versions is a really
bad idea. All releases should follow the strictly numeric
(Major.Minor.Patch) pattern that we've used for all of the releases except
the 2.0.x ones.

-- Owen


On Mon, Feb 4, 2013 at 11:53 AM, Stack st...@duboce.net wrote:

 On Mon, Feb 4, 2013 at 10:46 AM, Arun C Murthy a...@hortonworks.com
 wrote:

  Would it better to have 2.0.3-alpha, 2.0.4-beta and then make 2.1 as a
  stable release? This way we just have one series (2.0.x) which is not
  suitable for general consumption.
 
 

 That contains the versioning damage to the 2.0.x set.  This is an
 improvement over the original proposal where we let the versioning mayhem
 run out 2.3.

 Thanks Arun,
 St.Ack

[jira] [Resolved] (HDFS-4009) WebHdfsFileSystem and HftpFileSystem don't need delegation tokens

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-4009.
-

Resolution: Won't Fix

This is a feature, not a bug.

In particular, if your kerberos ticket has 1 hour left, the application will 
fail without a token. In tools that copy large amounts of data using the http 
filesystem, this happens relatively often.

 WebHdfsFileSystem and HftpFileSystem don't need delegation tokens
 -

 Key: HDFS-4009
 URL: https://issues.apache.org/jira/browse/HDFS-4009
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Tom White
Assignee: Karthik Kambatla
 Attachments: hadoop-8852.patch, hadoop-8852.patch, 
 hadoop-8852-v1.patch


 Parent JIRA to track the work of removing delegation tokens from these 
 filesystems. 
 This JIRA has evolved from the initial issue of these filesystems not 
 stopping the DelegationTokenRenewer thread they were creating.
 After further investigation, Daryn pointed out - If you can get a token, you 
 don't need a token! Hence, these filesystems shouldn't use delegation tokens.
 Evolution of the JIRA is listed below:
 Update 2:
 DelegationTokenRenewer is not required. The filesystems that are using it 
 already have Krb tickets and do not need tokens. Remove 
 DelegationTokenRenewer and all the related logic from WebHdfs and Hftp 
 filesystems.
 Update1:
 DelegationTokenRenewer should be Singleton - the instance and renewer threads 
 should be created/started lazily. The filesystems using the renewer shouldn't 
 need to explicity start/stop the renewer, and only register/de-register for 
 token renewal.
 Initial issue:
 HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer 
 thread when they are closed. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-4010) Remove unused TokenRenewer implementation from WebHdfsFileSystem and HftpFileSystem

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-4010.
-

Resolution: Won't Fix

They are used via the java ServiceLoader interface.

 Remove unused TokenRenewer implementation from WebHdfsFileSystem and 
 HftpFileSystem
 ---

 Key: HDFS-4010
 URL: https://issues.apache.org/jira/browse/HDFS-4010
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: HDFS-4010.patch


 WebHdfsFileSystem and HftpFileSystem implement TokenRenewer without using 
 anywhere.
 As we are in the process of migrating them to not use tokens, this code 
 should be removed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-4011) WebHdfsFileSystem shouldn't implicitly fetch delegation tokens

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-4011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-4011.
-

Resolution: Invalid

This is a feature.

 WebHdfsFileSystem shouldn't implicitly fetch delegation tokens
 --

 Key: HDFS-4011
 URL: https://issues.apache.org/jira/browse/HDFS-4011
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-4012) HftpFileSystem shouldn't implicity fetch delegation tokens

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-4012.
-

Resolution: Not A Problem

This is not a problem.

 HftpFileSystem shouldn't implicity fetch delegation tokens
 --

 Key: HDFS-4012
 URL: https://issues.apache.org/jira/browse/HDFS-4012
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Karthik Kambatla



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3993) The KSSL class should not limit the ssl ciphers

[Owen O'Malley (JIRA)]

Owen O'Malley created HDFS-3993:
---

 Summary: The KSSL class should not limit the ssl ciphers
 Key: HDFS-3993
 URL: https://issues.apache.org/jira/browse/HDFS-3993
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Owen O'Malley


The KSSL class' static block currently limits the ssl ciphers to a single 
value. It should use a much more permissive list.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3749) Disable check for jsvc on windows

[Owen O'Malley (JIRA)]

Owen O'Malley created HDFS-3749:
---

 Summary: Disable check for jsvc on windows
 Key: HDFS-3749
 URL: https://issues.apache.org/jira/browse/HDFS-3749
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: data-node
Reporter: Owen O'Malley
Assignee: Owen O'Malley


Jsvc doesn't make sense on windows and thus we should not require the datanode 
to start up under it on that platform.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-2386) with security enabled fsck calls lead to handshake_failure and hftp fails throwing the same exception in the logs

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-2386.
-

Resolution: Invalid

Fixed via HDFS-2617.

 with security enabled fsck calls lead to handshake_failure and hftp fails 
 throwing the same exception in the logs
 -

 Key: HDFS-2386
 URL: https://issues.apache.org/jira/browse/HDFS-2386
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 0.20.205.0
Reporter: Arpit Gupta



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3466) The SPNEGO filter for the NameNode should come out of the web keytab file

[Owen O'Malley (JIRA)]

Owen O'Malley created HDFS-3466:
---

 Summary: The SPNEGO filter for the NameNode should come out of the 
web keytab file
 Key: HDFS-3466
 URL: https://issues.apache.org/jira/browse/HDFS-3466
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node, security
Affects Versions: 2.0.0-alpha, 1.1.0
Reporter: Owen O'Malley
Assignee: Owen O'Malley


Currently, the spnego filter uses the DFS_NAMENODE_KEYTAB_FILE_KEY to find the 
keytab. It should use the DFS_WEB_AUTHENTICATION_KERBEROS_KEYTAB_KEY to do it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3461) HFTP should use the same port protocol for getting the delegation token

[Owen O'Malley (JIRA)]

Owen O'Malley created HDFS-3461:
---

 Summary: HFTP should use the same port  protocol for getting the 
delegation token
 Key: HDFS-3461
 URL: https://issues.apache.org/jira/browse/HDFS-3461
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Owen O'Malley
Assignee: Owen O'Malley
 Fix For: 1.1.0


Currently, hftp uses http to the Namenode's https port, which doesn't work.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3374) hdfs' TestDelegationToken fails intermittently with a race condition

[Owen O'Malley (JIRA)]

Owen O'Malley created HDFS-3374:
---

 Summary: hdfs' TestDelegationToken fails intermittently with a 
race condition
 Key: HDFS-3374
 URL: https://issues.apache.org/jira/browse/HDFS-3374
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Reporter: Owen O'Malley
Assignee: Owen O'Malley


The testcase is failing because the MiniDFSCluster is shutdown before the 
secret manager can change the key, which calls system.exit with no edit streams 
available.

{code}

[junit] 2012-05-04 15:03:51,521 WARN  common.Storage 
(FSImage.java:updateRemovedDirs(224)) - Removing storage dir 
/home/horton/src/hadoop/build/test/data/dfs/name1
[junit] 2012-05-04 15:03:51,522 FATAL namenode.FSNamesystem 
(FSEditLog.java:fatalExit(388)) - No edit streams are accessible
[junit] java.lang.Exception: No edit streams are accessible
[junit] at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.fatalExit(FSEditLog.java:388)
[junit] at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.exitIfNoStreams(FSEditLog.java:407)
[junit] at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.removeEditsAndStorageDir(FSEditLog.java:432)
[junit] at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.removeEditsStreamsAndStorageDirs(FSEditLog.java:468)
[junit] at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:1028)
[junit] at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.logUpdateMasterKey(FSNamesystem.java:5641)
[junit] at 
org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenSecretManager.logUpdateMasterKey(DelegationTokenSecretManager.java:286)
[junit] at 
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.updateCurrentKey(AbstractDelegationTokenSecretManager.java:150)
[junit] at 
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.rollMasterKey(AbstractDelegationTokenSecretManager.java:174)
[junit] at 
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager$ExpiredTokenRemover.run(AbstractDelegationTokenSecretManager.java:385)
[junit] at java.lang.Thread.run(Thread.java:662)
[junit] Running org.apache.hadoop.hdfs.security.TestDelegationToken
[junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
[junit] Test org.apache.hadoop.hdfs.security.TestDelegationToken FAILED 
(crashed)
{code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3348) After HDFS-2617 can't use 0.0.0.0 in dfs.http.address

[Owen O'Malley (JIRA)]

Owen O'Malley created HDFS-3348:
---

 Summary: After HDFS-2617 can't use 0.0.0.0 in dfs.http.address
 Key: HDFS-3348
 URL: https://issues.apache.org/jira/browse/HDFS-3348
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Owen O'Malley


After HDFS-2617, if you use 0.0.0.0 in dfs.http.address, the _HOST resolution 
for SPNEGO will create a principal like nn/0.0@example.com.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3345) Primary and secondary Principals must be the same

[Owen O'Malley (JIRA)]

Owen O'Malley created HDFS-3345:
---

 Summary: Primary and secondary Principals must be the same
 Key: HDFS-3345
 URL: https://issues.apache.org/jira/browse/HDFS-3345
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Owen O'Malley


The NameNode and SecondaryNameNode have two different configuration knobs 
(dfs.namenode.kerberos.principal and 
dfs.secondary.namenode.kerberos.principal), but the secondary namenode fails 
authorization unless it is the same user.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3316) The tar ball doesn't include jsvc any more

[Owen O'Malley (JIRA)]

Owen O'Malley created HDFS-3316:
---

 Summary: The tar ball doesn't include jsvc any more
 Key: HDFS-3316
 URL: https://issues.apache.org/jira/browse/HDFS-3316
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Owen O'Malley
Assignee: Owen O'Malley
 Fix For: 1.0.3


The current release tarballs on the 1.0 branch don't include jsvc by default.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [RESULT] - [VOTE] Rename hadoop branches post hadoop-1.x

[Owen O'Malley]

On Wed, Mar 28, 2012 at 5:11 PM, Doug Cutting cutt...@apache.org wrote:

 On 03/28/2012 12:39 PM, Owen O'Malley wrote:
  [ ... ] So the RM of the 2 branch needs to make the call of what
  should be 2.1 vs 3.0.

 I thought these were community decisions, not RM decisions, no?


 What to release is the RM's decision and then voted on by the community.
We tried voting on which features to include and it led to no releases for
two years. I think our users are better served by having good usable
releases.

-- Owen

Re: [RESULT] - [VOTE] Rename hadoop branches post hadoop-1.x

[Owen O'Malley]

I disagree. Trunk should become branch-3 once someone wants to start
stabilizing it. Arun is going to need the minor versions for when he adds
features.

X.Y.Z

Z = bug fixes
Y = minor release (compatible, adds features)
X = major release (incompatible)

So from branch-2 will come branch-2.0 with tags for 2.0.0, 2.0.1. New
features will go into branch-2, which will become branch-2.1, branch-2.2,
and so on.

-- Owen

Re: [RESULT] - [VOTE] Rename hadoop branches post hadoop-1.x

[Owen O'Malley]

On Wed, Mar 28, 2012 at 12:32 PM, Todd Lipcon t...@cloudera.com wrote:

But new features also go to trunk. And if none of our new features are
 incompatible, why do we anticipate that trunk is 3.0?


Let's imagine that we already had a 2.0.0 release. Now we want to add
features like HA. The only place to put that is in 2.1.0. On the other
hand, you don't want to pull *ALL* of the changes from trunk. That is way
too much scope. So the RM of the 2 branch needs to make the call of what
should be 2.1 vs 3.0.

-- Owen

Re: svn commit: r1304067 - in /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project: ./ bin/ conf/ hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/ hadoop-mapreduce-exam

[Owen O'Malley]

To me, I'd much much rather have the human readable description of what is
being fixed and I mostly could care less about which subversion commit it
corresponds to. I'd be all for using the CHANGE.txt description as the
commit message for both trunk and the branches.

-- Owen

[jira] [Created] (HDFS-2856) Fix block protocol so that Datanodes don't require root or jvsc

[Owen O'Malley (Created) (JIRA)]

Fix block protocol so that Datanodes don't require root or jvsc
---

 Key: HDFS-2856
 URL: https://issues.apache.org/jira/browse/HDFS-2856
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: contrib/raid, scripts
Reporter: Owen O'Malley


Since we send the block tokens unencrypted to the datanode, we currently start 
the datanode as root and get a secure ( 1024) port.

If we have the datanode generate a nonce and send it on the connection and the 
sends an hmac of the nonce back instead of the block token it won't reveal any 
secrets. Thus, we wouldn't require a secure port and would not require root or 
jsvc.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-224) I propose a tool for creating and manipulating a new abstraction, Hadoop Archives.

[Owen O'Malley (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-224.


Resolution: Duplicate

We have a different version of harchives.

 I propose a tool for creating and manipulating a new abstraction, Hadoop 
 Archives.
 --

 Key: HDFS-224
 URL: https://issues.apache.org/jira/browse/HDFS-224
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Dick King

 -- Introduction
 In some hadoop map/reduce and dfs use cases, including a specific case that 
 arises in my own work, users would like to populate dfs with a family of 
 hundreds or thousands of directory trees, each of which consists of thousands 
 of files.  In our case, the trees each have perhaps 20 gigabytes; two or 
 three 3-10-gigabyte files, a thousand small ones, and a large number of files 
 of intermediate size.  I am writing this JIRA to encourage discussion of a 
 new facility I want to create and contribute to the dfs core.
 -- The problem
 You can't store such families of trees in dfs in the obvious manner.  The 
 problem is that the name nodes can't handle the millions or ten million files 
 that result from such a family, especially if there are a couple of families. 
  I understand that dfs will not be able to accommodate tens of millions of 
 files in one instance for quite a while.
 -- Exposed API of my proposed solution
 I would therefore like to produce, and contribute to the dfs core, a new tool 
 that implements an abstraction called a Hadoop Archive [or harchive].  
 Conceptually, a harchive is a unit, but it manages a space that looks like a 
 directory tree.  The tool exposes an interface that allows a user to do the 
 following:
  * directory-level operations
** create a harchive [either empty, or initially populated form a 
 locally-stored directory tree] .  The namespace for harchives is the same as 
 the space of possible dfs directory locators, and a harchive would in fact be 
 implemented as a dfs directory with specialized contents.
** Add a directory tree to an existing harchive in a specific place within 
 the harchive
** retrieve a directory tree or subtree at or beneath the root of the 
 harchive directory structure, into a local directory tree
  * file-level operations
** add a local file to a specific place in the harchive
** modify a file image in a specific place in the harchive to match a 
 local file
** delete a file image in the harchive.
** move a file image within the harchive
** open a file image in the harchive for reading or writing.
  * stream operations
** open a harchive file image for reading or writing as a stream, in a 
 manner similar to dfs files, and read or write it [ie., hdfsRead(...) ].  
 This would include random access operators for reading.
  * management operations
** commit a group of changes [which would be made atomically -- there 
 would be no way half of a change could be made to a harchive if a client 
 crashes].
** clean up a harchive, if it's gotten less performant because of 
 extensive editing
** delete a harchive
 We would also implement a command line interface.
 -- Brief sketch of internals
 A harchive would be represented as a small collection of files, called 
 segments, in a dfs directory at the harchive's location.  Each segment would 
 contain some of the files of the harchive's file images in a format to be 
 determined, plus a harchive index.  We may group files by size, or some other 
 criteria.  It is likely that harchives would contain only one segment in 
 common cases.
 Changes would be made by adding the text of the new files, either by 
 rewriting an existing segment that contains not much more data than the size 
 of the changes or by creating a new segment, complete with a new index.  When 
 dfs comes to be enhanced to allow appends to dfs files, as requested by 
 HADOOP-1700 , we would be able to take advantage of that.
 Often, when a harchive is initially populated, it could be a single segment, 
 and a file it contains could be accessed with two random accesses into the 
 segment.  The first access retrieves the index, and the second access 
 retrieves the beginning of the file.  We could choose to put smaller files 
 closer to the index to allow lower average amortized costs per byte.
 We might instead choose to represent a harchive as one file or a few files 
 for the large represented files, and smaller files for the represented 
 smaller files.  That lets us make modifications by copying at lower cost.
 The segment containing the index is found by a naming convention.  Atomicity 
 is obtained by creating indices and renaming the files containing them 
 according to the convention, when a change is committed

[jira] [Created] (HDFS-2619) Remove my personal email address from the libhdfs build file.

[Owen O'Malley (Created) (JIRA)]

Remove my personal email address from the libhdfs build file.
-

 Key: HDFS-2619
 URL: https://issues.apache.org/jira/browse/HDFS-2619
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: build
Reporter: Owen O'Malley
Assignee: Owen O'Malley


When I first wrote the libhdfs autoconf/automake stuff, I incorrectly put my 
email address in the AC_INIT line, which means if something goes wrong, you get:

{quote}
 configure: WARNING:     ## Report this to my-email ##
{quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Retire 0.20-append branch?

[Owen O'Malley]

On Mon, Oct 3, 2011 at 2:50 PM, Todd Lipcon t...@cloudera.com wrote:
 In essence, that means that we would:
 1) Ask contributors to provide patches against 0.20-security branch
 instead of 0.20-append

+1

 2) Move the branch itself into some kind of attic/ directory

You can just delete it. Subversion always keeps branches from previous
points in time.

-- Owen

[jira] [Resolved] (HDFS-2328) hftp throws NPE if security is not enabled on remote cluster

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-2328.
-

Resolution: Fixed

I committed this to 20-s and 205. I'll commit it to trunk as part of 
MAPREDUCE-2764.

 hftp throws NPE if security is not enabled on remote cluster
 

 Key: HDFS-2328
 URL: https://issues.apache.org/jira/browse/HDFS-2328
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 0.20.205.0
Reporter: Daryn Sharp
Assignee: Owen O'Malley
Priority: Critical
 Fix For: 0.20.205.0

 Attachments: h-2328.patch


 If hftp cannot locate either a hdfs or hftp token in the ugi, it will call 
 {{getDelegationToken}} to acquire one from the remote nn.  This method may 
 return a null {{Token}} if security is disabled(*)  on the remote nn.  Hftp 
 will internally call its {{setDelegationToken}} which will throw a NPE when 
 the token is {{null}}.
 (*) Actually, if any problem happens while acquiring the token it assumes 
 security is disabled!  However, it's a pre-existing issue beyond the scope of 
 the token renewal changes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-1952) FSEditLog.open() appears to succeed even if all EDITS directories fail

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-1952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-1952.
-

Resolution: Fixed

Resolving this, since it was committed to trunk.

 FSEditLog.open() appears to succeed even if all EDITS directories fail
 --

 Key: HDFS-1952
 URL: https://issues.apache.org/jira/browse/HDFS-1952
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 0.22.0, 0.23.0
Reporter: Matt Foley
Assignee: Andrew Wang
  Labels: newbie
 Attachments: hdfs-1952-0.22.patch, hdfs-1952.patch, hdfs-1952.patch, 
 hdfs-1952.patch


 FSEditLog.open() appears to succeed even if all of the individual 
 directories failed to allow creation of an EditLogOutputStream.  The problem 
 and solution are essentially similar to that of HDFS-1505.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-1666) TestAuthorizationFilter is failing

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-1666.
-

   Resolution: Fixed
Fix Version/s: 0.22.0
 Assignee: Todd Lipcon

This was committed to trunk.

 TestAuthorizationFilter is failing
 --

 Key: HDFS-1666
 URL: https://issues.apache.org/jira/browse/HDFS-1666
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: contrib/hdfsproxy
Affects Versions: 0.22.0, 0.23.0
Reporter: Konstantin Boudnik
Assignee: Todd Lipcon
Priority: Blocker
 Fix For: 0.22.0

 Attachments: hdfs-1666-disable-tests.txt


 two test cases were failing for a number of builds (see attached logs)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-1809) Lack of proper DefaultMetricsSystem initialization breaks some tests

[Owen O'Malley (JIRA)]

Lack of proper DefaultMetricsSystem initialization breaks some tests


 Key: HDFS-1809
 URL: https://issues.apache.org/jira/browse/HDFS-1809
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Owen O'Malley
Assignee: Suresh Srinivas
 Fix For: 0.20.203.0


Following tests are failing:
TestHDFSServerPorts
TestNNLeaseRecovery
TestSaveNamespace

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-1810) Remove duplicate jar entries from common

[Owen O'Malley (JIRA)]

Remove duplicate jar entries from common


 Key: HDFS-1810
 URL: https://issues.apache.org/jira/browse/HDFS-1810
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Owen O'Malley
Assignee: Luke Lu


Remove the jars that we get from common from our direct dependency list.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-1811) Create scripts to decommission datanodes

[Owen O'Malley (JIRA)]

Create scripts to decommission datanodes


 Key: HDFS-1811
 URL: https://issues.apache.org/jira/browse/HDFS-1811
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Owen O'Malley
Assignee: Erik Steffl


Create scripts to decommission datanodes:

  - distribute exclude file
- input is location of exclude file
- location on namenodes: hdfs getconf -excludeFile
- list of namenodes: hdfs getconf -namenodes
- scp excludes files to all namenodes

  - refresh namenodes
- list of namenodes: hdfs getconf -namenodes
- refresh namenodes: hdfs dfsadmin -refreshNodes

Two scripts are needed because each of them might require different permissions.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Created: (HDFS-1729) Improve metrics for measuring NN startup costs.

[Owen O'Malley (JIRA)]

Improve metrics for measuring NN startup costs.
---

 Key: HDFS-1729
 URL: https://issues.apache.org/jira/browse/HDFS-1729
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: name-node
Reporter: Owen O'Malley
Assignee: Matt Foley
 Fix For: 0.20.100


Current logging and metrics are insufficient to diagnose latency problems in 
cluster startup.  Add:
1. better logs in both Datanode and Namenode for Initial Block Report 
processing, to help distinguish between block
report processing problems and RPC/queuing problems;
2. new logs to measure cost of scanning all blocks for over/under/invalid 
replicas, which occurs in Namenode just
before exiting safe mode;
3. new logs to measure cost of processing the under/invalid replica queues 
(created by the above mentioned scan), which
occurs just after exiting safe mode, and is said to take 100% of CPU.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: hadoop.job.ugi backwards compatibility

[Owen O'Malley]

On Mon, Sep 13, 2010 at 11:10 AM, Todd Lipcon t...@cloudera.com wrote:
 Yep, but there are plenty of 10 node clusters out there that do important
 work at small startups or single-use-case installations, too. We need to
 provide scalability and security features that work for the 100+ node
 clusters but also not leave the beginners in the dust.

10 node clusters are an important use case, but creating the user
accounts on those clusters is very easy because of the few users.
Futhermore, if the accounts aren't there it just means the users have
no groups. Which for a single use system with security turned off
isn't the end of the world.

 But I think there are plenty of people out there who have built small
 webapps, shell scripts, cron jobs, etc that use hadoop.job.ugi on some
 shared account to impersonate other users.

I'd be surprised. At Yahoo, the primary problem came with people
screen scraping the jobtracker http. With security turned off that
isn't an issue. Again, it isn't hard, just the evolving interface of
UserGroupInformation changed. With security, we tried really hard to
maintain backwards compatibility and succeeded for the vast (99%+)
majority of the users.

 Perhaps I am estimating
 incorrectly - that's why I wanted this discussion on a user-facing list
 rather than a dev-facing list.

Obviously the pointer is there for them to follow into the rabbit hole
of the dev lists. *grin*

 Another example use case that I do a lot on non-secure clusters is: hadoop
 fs -Dhadoop.job.ugi=hadoop,hadoop something I want to do as a superuser.
 The permissions model we have in 0.20 obviously isn't secure, but it's nice
 to avoid accidental mistakes, and making it easy to sudo like that is
 handy.

It might make sense to add a new switch ( -user ?) to hadoop fs that
does a doAs before doing the
shell command. You could even make it fancy and try to be a proxy user
if security is turned on.

 Regardless of our particular opinions, isn't our policy that we cannot break
 API compatibility between versions without a one-version deprecation period?

There wasn't a way to keep UGI stable. It was a broken design before
the security work. It is marked evolving so we try to minimize
breakage, but it isn't prohibited.

-- Owen

[jira] Resolved: (HDFS-98) creating a file in hdfs should not automatically create the parent directories

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-98.
---

Resolution: Won't Fix

Leaving the old FileSystem behavior and fixing it in FileContext is right. 
Closing this.

 creating a file in hdfs should not automatically create the parent directories
 --

 Key: HDFS-98
 URL: https://issues.apache.org/jira/browse/HDFS-98
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Owen O'Malley
Assignee: Pi Song
 Attachments: hadoop-2759-complete1.patch, HADOOP-2759_1.patch, 
 hadoop_tmp.patch


 I think it would be better if HDFS didn't automatically create directories 
 for the user. In particular, in clean up code, it would be nice if deleting a 
 directory couldn't be undone by mistake by a process that hasn't been killed 
 yet creating a new file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: Integrating Lustre and HDFS

[Owen O'Malley]

 Your local storage should get used for MR.  Use Lustre via file:// 
 (LocalFileSystem, iirc)
 instead of HDFS via hdfs:// (DistributedFileSystem, irrc) as the default file 
 system type.

If Lustre has integrated checksums, you'll want to use the
RawLocalFileSystem instead of LocalFileSystem. You'll want to make it
accessible via:

fs.raw.impl = org.apache.hadoop.fs.RawLocalFileSystem

so that urls like raw:///my/path won't go through the Hadoop checksum code.

-- Owen

[jira] Created: (HDFS-1187) Modify fetchdt to allow renewing and canceling token

[Owen O'Malley (JIRA)]

Modify fetchdt to allow renewing and canceling token


 Key: HDFS-1187
 URL: https://issues.apache.org/jira/browse/HDFS-1187
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: tools
Reporter: Owen O'Malley
Assignee: Owen O'Malley
 Attachments: fetchdt.patch

I would like to extend fetchdt to allow renewing and canceling tokens.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (HDFS-1178) The NameNode servlets should not use RPC to connect to the NameNode

[Owen O'Malley (JIRA)]

The NameNode servlets should not use RPC to connect to the NameNode
---

 Key: HDFS-1178
 URL: https://issues.apache.org/jira/browse/HDFS-1178
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Reporter: Owen O'Malley
Assignee: Owen O'Malley


Currently some of the NameNode servlets use RPC to connect from the NameNode to 
itself. They should do it more directly with the NameNode object.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (HDFS-986) Push HADOOP-6551 into HDFS

[Owen O'Malley (JIRA)]

Push HADOOP-6551 into HDFS
--

 Key: HDFS-986
 URL: https://issues.apache.org/jira/browse/HDFS-986
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Owen O'Malley


We need to throw readable error messages instead of returning false on errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (HDFS-787) Make the versions of libraries consistent

[Owen O'Malley (JIRA)]

Make the versions of libraries consistent
-

 Key: HDFS-787
 URL: https://issues.apache.org/jira/browse/HDFS-787
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Owen O'Malley
Assignee: Owen O'Malley
Priority: Blocker
 Fix For: 0.21.0, 0.22.0


This is the version for HDFS.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (HDFS-641) Move all of the benchmarks and tests that depend on mapreduce to mapreduce

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-641.


Resolution: Fixed

I just committed this.

 Move all of the benchmarks and tests that depend on mapreduce to mapreduce
 --

 Key: HDFS-641
 URL: https://issues.apache.org/jira/browse/HDFS-641
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Affects Versions: 0.20.2
Reporter: Owen O'Malley
Assignee: Owen O'Malley
Priority: Blocker
 Fix For: 0.21.0


 Currently, we have a bad cycle where to build hdfs you need to test mapreduce 
 and iterate once. This is broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Post-5107 cross-project commits

[Owen O'Malley]

I've committed HADOOP-5107 to Common and HDFS. MapReduce will follow  
soon.


There are a couple of things that change:

1. The non-release version names move to the Maven-friendly 0.22.0- 
SNAPSHOT instead of 0.22.0-dev.

2. To change Common and push it to your local Maven repository:
  a. make change to common
  b. ant mvn-install
3. To build using your internal Maven repository and push HDFS to your  
local Maven repository:

  a. make change to HDFS
  b. ant -Dresolvers=internal clean-cache mvn-install

-- Owen

Welcome Konstantin Boudnik as a qa committer!

[Owen O'Malley]

The Hadoop PMC has voted to make Cos a QA committer on Common, HDFS,  
and MapReduce. I'd like to welcome Cos as the newest committer.


-- Owen

[jira] Resolved: (HDFS-638) The build.xml refences jars that don't exist

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley resolved HDFS-638.


Resolution: Fixed

Suresh fixed this without a jira.

 The build.xml refences jars that don't exist
 

 Key: HDFS-638
 URL: https://issues.apache.org/jira/browse/HDFS-638
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
Reporter: Owen O'Malley
Assignee: Owen O'Malley
 Fix For: 0.22.0


 Currently the build is broken on trunk. The jar files need to be updated to 
 the current ones, along with the new version.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (HDFS-641) Move all of the benchmarks and tests that depend on mapreduce to mapreduce

[Owen O'Malley (JIRA)]

Move all of the benchmarks and tests that depend on mapreduce to mapreduce
--

 Key: HDFS-641
 URL: https://issues.apache.org/jira/browse/HDFS-641
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Affects Versions: 0.20.2
Reporter: Owen O'Malley
Assignee: Owen O'Malley
Priority: Blocker
 Fix For: 0.21.0


Currently, we have a bad cycle where to build hdfs you need to test mapreduce 
and iterate once. This is broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.