Add support for writing fractional seconds

[belegdol]

This is initial work towards implementing issue #336. The current status is:

• pandas backend with pandas-3.0.3 passes tests
• polars backend with pandas-3.0.3 fails the fractional seconds test because the datetime is still truncated to whole seconds. I need to find out why.
• pandas-2.3.3 fails because it tries to use the vectorised function on the datetime object column which obviously fails. I am not sure why that is.

[ofajardo]

Regarding the problem with Polars, this fixes it for me (this is the git diff output on _readstat_writer.pyx, so - means I removed that line and + I added a line). Pandas tests also pass for pandas 3 (Have not checked the pandas 2 issue yet). Please give it a try.

import datetime
 #import calendar
 from datetime import timezone
-from decimal import Decimal
 #from datetime import timezone as _timezone
 #from libc.math cimport round, NAN

@@ -88,9 +87,11 @@ cdef object vectorized_convert_datetime_to_number(object df, dst_file_format fil
     df = df.with_columns(nw.nth(col_indxs).cast(nw.Int64))
     for col_indx in col_indxs:
         convfac = convfacs[pywriter_timeunits[col_indx]]
-        finfac = Decimal(mulfac) / Decimal(convfac)
+        total = nw.nth(col_indx) + offset_secs * convfac
+        whole = total // convfac
+        frac = total % convfac
         df = df.with_columns(nw.when(nw.nth(col_indx) != -9223372036854775808).then(
-            ((nw.nth(col_indx) + offset_secs * convfac) * finfac).cast(nw.Float64)))
+            (whole.cast(nw.Float64) * mulfac + frac.cast(nw.Float64) / convfac * mulfac)))
     return df

[ofajardo]

and this one fixes the pandas 2.3.3 issue, both pandas and polars tests passing, also upgrading again to pandas 3 works: (again gitdiff on _readstat_writer.pyx)

@@ -87,9 +87,8 @@ cdef object vectorized_convert_datetime_to_number(object df, dst_file_format fil
     df = df.with_columns(nw.nth(col_indxs).cast(nw.Int64))
     for col_indx in col_indxs:
         convfac = convfacs[pywriter_timeunits[col_indx]]
-        total = nw.nth(col_indx) + offset_secs * convfac
-        whole = total // convfac
-        frac = total % convfac
+        whole = nw.nth(col_indx) // convfac + offset_secs
+        frac = nw.nth(col_indx) % convfac
         df = df.with_columns(nw.when(nw.nth(col_indx) != -9223372036854775808).then(
             (whole.cast(nw.Float64) * mulfac + frac.cast(nw.Float64) / convfac * mulfac)))
     return df

[belegdol]

Nice! Now the tests pass with pandas-2.3.3, pandas-3.0.3 and polars. Would you like to commit the changes since it was your idea?

[ofajardo]

to make it easy, if you have it in your fork and working, just go ahead and commit, or you can also add me as co-author:, something like that: https://gist.github.com/lisawolderiksen/f9747a3ae1e58e9daa7d176ab98f1bad
https://docs.github.com/en/pull-requests/committing-changes-to-your-project/creating-and-editing-commits/creating-a-commit-with-multiple-authors

[ofajardo]

Let me know once you think this is ready for merging

[belegdol]

I am not sure what else to test. I could try reading the generated xpt in sas and exporting it as csv to see that it matches the source csv.

[belegdol]

SAS seems to like the files too. I used the following code, once for pandas-3.x and once for polars:

%xpt2loc(FILESPEC='fractional_seconds.xpt');

proc datasets;
modify dataset;
	format date e8601da10. dtime e8601dt26.6 time e8601tm15.6;
run;

proc export data=dataset outfile="fractional_seconds.csv";
run;

The re-exported files match test_data/basic/fractional_seconds.csv exactly.
Unless you have ideas what else to test, I would say this is as ready for merging as it is going to get.