dataset write api

[dbelozerovx1]

why is dataset is designed to write a single parquet file, i mean it throws an error if write to non empty directory into s3, but usually tables consists of many files, am i did something wrong? but i do not see an option to write multiple parquet files into a directory using dataset

[amoeba]

Hi @dbelozerovx1, arrow-java's Dataset implementation supports writing a dataset out to multiple files like you expect, see https://github.com/apache/arrow-java/blob/main/dataset/src/test/java/org/apache/arrow/dataset/file/TestDatasetFileWriter.java#L66 for an example. If you want to share the code you have that isn't working, I can take a look.