You can also load multiple adm files into a same dataset with a single AQL
as follows:
load dataset Tweets
using "org.apache.asterix.external.dataset.adapter.NCFileSystemAdapter"
(("path"=
"130.149.249.60
:///data/seok.kim/spatial-index-experiment/files/SyntheticTweetsRectangleHouse200M-psi27-pid0.adm,
130.149.249.53
:///data/seok.kim/spatial-index-experiment/files/SyntheticTweetsRectangleHouse200M-psi26-pid1.adm,
130.149.249.54
:///data/seok.kim/spatial-index-experiment/files/SyntheticTweetsRectangleHouse200M-psi25-pid2.adm,
130.149.249.55
:///data/seok.kim/spatial-index-experiment/files/SyntheticTweetsRectangleHouse200M-psi24-pid3.adm,
130.149.249.56
:///data/seok.kim/spatial-index-experiment/files/SyntheticTweetsRectangleHouse200M-psi23-pid4.adm,
130.149.249.57
:///data/seok.kim/spatial-index-experiment/files/SyntheticTweetsRectangleHouse200M-psi22-pid5.adm,
130.149.249.58
:///data/seok.kim/spatial-index-experiment/files/SyntheticTweetsRectangleHouse200M-psi21-pid6.adm,
130.149.249.59
:///data/seok.kim/spatial-index-experiment/files/SyntheticTweetsRectangleHouse200M-psi20-pid7.adm"),
("format"="adm"));
The above AQL loads 8 adm files into a single dataset named Tweets.
Cheers,
Young-Seok
On Fri, Mar 4, 2016 at 12:19 PM, Xikui Wang <[email protected]> wrote:
> Hi Yingyi,
>
> Thanks for your reply. I think the external dataset with scan query is a
> good solution.
> I will try that. Thank you.
>
> Best,
> Xikui
>
> On Fri, Mar 4, 2016 at 11:53 AM, Yingyi Bu <[email protected]> wrote:
>
> > Xikui,
> >
> > If the number of localfs files is too large, a solution could be to put
> > your files on HDFS and then load it. Loading from HDFS always has a
> fixed
> > degree of parallelism regardless of the number of files.
> >
> > >> I am wondering is there a way to append adm file to existed dataset?
> > You can create an external dataset and then write an insert statement
> where
> > the body is a scan query. AsterixDB doesn't load any data into its own
> > storage for an external dataset but just keeps file paths.
> > Here is a manual for external datasets:
> > https://ci.apache.org/projects/asterixdb/aql/externaldata.html
> >
> > Best,
> > Yingyi
> >
> >
> > On Fri, Mar 4, 2016 at 11:47 AM, Xikui Wang <[email protected]> wrote:
> >
> > > Hi,
> > >
> > > I want to import data from multiple adm files into a same dataset.
> > Merging
> > > them together and then loading from localfs can be a viable solution,
> but
> > > this may become a problem when the number become too large. I am
> > wondering
> > > is there a way to append adm file to existed dataset?
> > >
> > > Thank you.
> > >
> > > Best,
> > > Xikui
> > >
> >
>
