80% of ML Newbies Struggle with Data Scaling in Palantir Foundry

In machine learning, scaling test data correctly is crucial. A common practice is to use the scaling parameters, like mean and standard deviation, from the training data to transform the test data. This ensures consistency in data preprocessing. However, a new user of Palantir Foundry Model Training has found that the platform does not provide an explicit option in its Evaluation Configuration to carry over these parameters from training to test data. This leaves many, especially those new to machine learning, unsure if Palantir automatically handles this process or if there’s a manual workaround needed. The user suggests using “fit_transform” for training data and “transform” for test data, but questions how to implement this in Palantir Foundry. This issue highlights a potential gap in user guidance or functionality within the platform for data scaling in model evaluation.

Source: stackoverflow.com