India Crop Yield Prediction

What this research found

Two Random Forest models were built on Indian agricultural data: one predicting crop yield from 237,932 district-season-crop records spanning 1997 to 2015, the other classifying soil samples as fertile or not from 16 measured properties. Using only state, season, crop and year, the yield model explains 71.0% of variance on held-out data at an RMSE of 14.46 tonnes per hectare, and analysis of variance confirms that yields differ significantly across both seasons and states.

• The yield regressor reached a test R² of 0.7103 with an RMSE of 14.4630 on 47,587 held-out records, against a training R² of 0.9179. The analysis flags that gap as overfitting.
• Year dominates the yield model at 0.4853 of feature importance, ahead of state at 0.3382 and crop at 0.1496. Season contributes almost nothing, at 0.0269.
• Yields differ significantly across the six seasons (F = 1264.82, with a p-value below floating-point resolution) and across the top five producing states (F = 112.28, p = 1.0985e-95). Tamil Nadu leads on mean yield at 12.26 tonnes per hectare, followed by Kerala at 8.62 and Assam at 8.57, against a national mean of 5.01.
• The soil fertility classifier scored 100% accuracy with perfect precision and recall on both classes — but on just 20 test samples drawn from a 100-record dataset.
• Texture outweighs nutrients in predicting the fertility label: clay content carries 0.2380 of importance and cation exchange capacity 0.1487, ahead of every macronutrient, with phosphorus highest at 0.0540. Nitrogen and organic carbon move together closely (r = 0.875), while pH and phosphorus are essentially unrelated (r = -0.104).

How it was done

A district-wise crop production dataset compiled from Government of India agricultural statistics supplied 246,091 records covering 105 crops across 33 states from 1997 to 2015, of which 237,932 survived cleaning, with yield derived from production and cultivated area. A Random Forest regressor was trained on an 80/20 split using state, season, crop and year. A separate 100-sample soil dataset carrying 16 properties — pH, electrical conductivity, organic carbon and matter, macro- and micronutrients, sand, silt and clay fractions, calcium carbonate and cation exchange capacity — trained a Random Forest classifier for binary fertility. One-way ANOVA then tested yield differences across seasons and across the five largest producing states, and the results were compiled into an illustrated report.

Data sources

• District-wise crop production dataset compiled from Government of India agricultural statistics — 246,091 records, 105 crops, 33 states and union territories, 1997–2015
• Soil fertility research dataset — 100 samples with 16 properties, aligned with the parameters of India's Soil Health Card scheme

Figures from this analysis

Outputs produced

• crop_production_district.csv

  District-wise crop production data for India (246,091 raw records)

• soil_fertility_data.csv

  Soil fertility data with 16 soil health parameters (100 records)

• 01_verify_datasets.py

  Python script to verify downloaded datasets and analyze structure

• dataset_inventory.md

  Comprehensive inventory of acquired datasets with metadata, column descriptions, and data quality notes

• 02_preprocess_data.py

  Data preprocessing pipeline: cleaning, standardization, yield calculation, and anomaly filtering

• clean_crop_production.csv

  Cleaned crop production data with calculated Yield column (237,932 records)

• clean_soil_data.csv

  Cleaned soil fertility data with Soil_Class and Texture_Class columns (100 records)

• state_crop_yield_summary.csv

  Aggregated yield statistics by State and Crop (1,168 combinations)