Tutorial 1: Automated Refinement (with BGMN)

Tutorial 1: Automated Refinement (with BGMN)#

Dara provides a Python-based wrapper to the refinement software, BGMN, which implements a robust optimization algorithm that can refine automatically in most cases. This tutorial will show you how to interact with BGMN software and how to submit, adjust, and visualize your refinements.

You can download this tutorial project from here.

%pip install ipywidgets nbformat

from pathlib import Path

from dara.refine import do_refinement_no_saving

data = Path("tutorial_data")
cif_paths = list(data.glob("*.cif"))  # include all the cif files in the data folder

pattern_fn = "CaNi(PO3)4_800_240_Ca(OH)2_(NH4)2HPO4_NiO.xy"

Basic Refinement#

A one-line refinement with the default settings.

Do refinement#

By running do_refinement_no_saving, the refinement will be performed and the results will be printed out. There will be no BGMN refinement output folder saved on the disk.

The only two things you will need to feed into the system:

  • the path to the pattern. Currently, Dara only supports the xy, xrdml, raw formats.

  • a list of CIF file paths. The CIF will be used as the reference structure for the refinement.

refinement = do_refinement_no_saving(data / pattern_fn, cif_paths)

2026-01-03 20:24:50,677 WARNING dara.bgmn_worker BGMN executable not found. Downloading BGMN.

  0%|          | 0.00/1.59M [00:00<?, ?iB/s]

 99%|█████████▊| 1.57M/1.59M [00:00<00:00, 6.41MiB/s]

100%|██████████| 1.59M/1.59M [00:00<00:00, 6.33MiB/s]

Visualization#

You can call visualize to visualize the refinement results. The observed, calculated, and difference patterns will be plotted.

refinement.visualize()

Save the refinement plot#

Optionally, if you want to share the plot with others, you can save the plot by calling write_image or write_html in the plotly.Figure object returned by .visualize(). The plot will be saved on the disk.

refinement.visualize().write_html("tutorial_refinement.html")  # output the interactive html file to the disk
refinement.visualize().write_image("tutorial_refinement.png")  # output the png image to the disk

Extracting information from the refinement#

After finishing refinement, you can read information from the RefinementResult object. The object contains the following attributes:

  • lst_data: information about phases, metrics of refinement (from the .lst file in BGMN)

  • peak_data: the simulated peaks in the calculated pattern

  • plot_data: x (two-theta), y_obs, y_calc, y_bkg, contribution from each phase. This is mainly used for visualization.

For example, you can get Rwp from lst_data

f"The refinement has Rwp = {refinement.lst_data.rwp} %"

'The refinement has Rwp = 5.24 %'

You can also get the information about the lattice, weight fraction for the phase.

Some values are in a tuple, with first value as the value and the second value as the error.

phase_name = "CaNi(PO3)4_15_sym"
phase_result = refinement.lst_data.phases_results[phase_name]

gewicht = phase_result.gewicht
lattice_a = phase_result.a
lattice_b = phase_result.b
lattice_c = phase_result.c
lattice_alpha = phase_result.alpha
lattice_beta = phase_result.beta
lattice_gamma = phase_result.gamma

print(f"The lattice parameters of the phase {phase_name} are:\n" \
f"    a = {lattice_a} nm, b = {lattice_b} nm, c = {lattice_c} nm,\n"  \
f"    alpha = {lattice_alpha}, beta = {lattice_beta}, gamma = {lattice_gamma}\n")
print(f"The weight fraction of the phase {phase_name} is ({gewicht[0]} ± {gewicht[1]}) %")

The lattice parameters of the phase CaNi(PO3)4_15_sym are:
    a = (1.20707, 0.0001) nm, b = (0.869244, 9.2e-05) nm, c = (0.97907, 0.00011) nm,
    alpha = None, beta = (117.9527, 0.0055), gamma = None

The weight fraction of the phase CaNi(PO3)4_15_sym is (0.2162 ± 0.0019) %

Peak data is stored in a pandas DataFrame.

refinement.peak_data
2theta intensity b1 b2 h k l phase phase_idx
0 37.419910 367.178051 0.004434 6.435354e-10 1 1 1 NiO_225_sym 0
1 43.435091 559.693988 0.004434 1.139670e-09 2 0 0 NiO_225_sym 0
2 62.965868 271.241200 0.004434 4.532117e-09 2 2 0 NiO_225_sym 0
3 75.453773 95.990765 0.004434 8.554955e-09 3 1 1 NiO_225_sym 0
4 79.430774 70.716772 0.004434 1.017677e-08 2 2 2 NiO_225_sym 0
... ... ... ... ... ... ... ... ... ...
481 99.288509 0.046459 0.004889 1.651996e-08 2 8 2 CaNi(PO3)4_15_sym 1
482 102.264363 8.212658 0.004889 1.800964e-08 5 1 6 CaNi(PO3)4_15_sym 1
483 99.476618 11.579483 0.004889 1.661263e-08 11 1 -1 CaNi(PO3)4_15_sym 1
484 100.286022 7.091383 0.004889 1.701459e-08 8 4 2 CaNi(PO3)4_15_sym 1
485 102.421640 1.774795 0.004889 1.809025e-08 3 5 -8 CaNi(PO3)4_15_sym 1

486 rows × 9 columns

Export the refined structure#

You can also export the refined structure as a pymatgen.Structure object.

structure = refinement.export_structure("CaNi(PO3)4_15_sym")
print(structure.to("tutorial_refinement_refined_CaNi(PO3)4_15_sym.cif", symprec=1e-3))

# generated using pymatgen
data_CaNi(PO3)4
_symmetry_space_group_name_H-M   C2/c
_cell_length_a   12.07070000
_cell_length_b   8.69244000
_cell_length_c   9.79070000
_cell_angle_alpha   90.00000000
_cell_angle_beta   117.95270000
_cell_angle_gamma   90.00000000
_symmetry_Int_Tables_number   15
_chemical_formula_structural   CaNi(PO3)4
_chemical_formula_sum   'Ca4 Ni4 P16 O48'
_cell_volume   907.43028822
_cell_formula_units_Z   4
loop_
 _symmetry_equiv_pos_site_id
 _symmetry_equiv_pos_as_xyz
  1  'x, y, z'
  2  '-x, -y, -z'
  3  '-x, y, -z+1/2'
  4  'x, -y, z+1/2'
  5  'x+1/2, y+1/2, z'
  6  '-x+1/2, -y+1/2, -z'
  7  '-x+1/2, y+1/2, -z+1/2'
  8  'x+1/2, -y+1/2, z+1/2'
loop_
 _atom_site_type_symbol
 _atom_site_label
 _atom_site_symmetry_multiplicity
 _atom_site_fract_x
 _atom_site_fract_y
 _atom_site_fract_z
 _atom_site_occupancy
  Ca  Ca0  4  0.00000000  0.04710000  0.25000000  1.0
  Ni  Ni1  4  0.25000000  0.25000000  0.50000000  1.0
  P  P2  8  0.00320000  0.72360000  0.47450000  1.0
  P  P3  8  0.18860000  0.49180000  0.19450000  1.0
  O  O4  8  0.03010000  0.86190000  0.41040000  1.0
  O  O5  8  0.06150000  0.25470000  0.42910000  1.0
  O  O6  8  0.07200000  0.60250000  0.15680000  1.0
  O  O7  8  0.13280000  0.36240000  0.06680000  1.0
  O  O8  8  0.21690000  0.08020000  0.33440000  1.0
  O  O9  8  0.22930000  0.42260000  0.34390000  1.0

Refining with customized phase parameters#

The refinement with default setting looks good. But can it be better?

Dara supports the basic refinement parameters in BGMN / Profex. You can adjust the refinement parameters by passing the parameters to the do_refinement_no_saving function.

Common parameters include:

  • lattice_range: you can (and need to) specify the range that the lattice parameters can vary. Usually, it can be a small range, like 0.01 ~ 0.05. It is applied to all lattice parameters (a, b, c, alpha, beta, gamma).

  • b1: controls the width of the peak. b1 describes the average particle size in the XRD sample. The larger the b1, the broader the peak. Usually, it is constraint to a small range, like from 0 to 0.005. If the fitted b1 is too large, you will see the peaks go too broad. In this case, your simulated pattern will look like an amorphous material that can be easily fit into the background.

  • k1: also controls the width of the peak. k1 describes the width of the particle size distribution in the sample. The larger the k1, the smaller the distribution is. Usually, it can be constrained to 0 ~ 1.

  • k2: describes the microstrain in the sample. The larger the k2, the larger the microstrain. Usually, it can be a fixed value, like 0.

  • gewicht: means “weight” in German. it contains the information of scale factor. However, in BGMN, it can also be used to specify the preferred orientation you would like to use in the refinement. By specifying the preferred orientation, you can vary the intensity of a set of reflections in the pattern, which can help you fit your pattern better. BGMN is able to decide which reflection to adjust automatically. You only need to specify how strong the preferred orientation is. Usually, it can be SPHAR0 (none), SPHAR2 (two preferred orientation parameters), or SPHAR4 (four preferred orientation parameters), … (up to SPHAR10). The larger the gewicht, the stronger the preferred orientation is. But it can cause overfitting as well.

Input parameter format#

In Dara, all the phase parameters are passed as a dictionary arg called phase_params. The key is the parameter name, and the value is the parameter value. Dara supports three types of values:

  • fixed. This is a string. The parameter will be fixed to the default value (usually 0).

  • (initial value)_(min value)^(max value). This is a string. The parameter will be allowed to vary in the refinement between initial value to the min value to the max value. The min value begins with _, and the max value begins with ^.

  • Other values. It can be a string or a number. For example, setting gewicht to SPHAR2 means that preferred orientation is modeled with the SPHAR2 settings; setting lattice_range to 0.05 means that the lattice parameters can vary up to 5%.. See the BGMN / Profex manual for more information on these settings.

If you want to allow 5% variation in lattice parameters, then use the following settings. b1: (started from 0, min = 0, max = 0.005), k1: (started from 0, min = 0, max = 1), k2: fixed to 0, and gewicht: SPHAR2. This corresponds to a phase_params dict of:

phase_params = {
    "lattice_range": 0.05,
    "b1": "0_0^0.005",
    "k1": "0_0^1",
    "k2": "fixed",
    "gewicht": "SPHAR2"
}

refinement = do_refinement_no_saving(data / pattern_fn, cif_paths, phase_params=phase_params)

refinement.visualize()

Now you can see the Rwp of the refinement is slightly lower, indicating a better fit.

Specify different parameters for different phases#

In the previous example, the refinement option is applied to all phases. But you can also specify different parameters for different phases. To do so, you will need to pass a special RefinementPhase object to the phases parameter.

from dara import RefinementPhase

phases = [RefinementPhase.make(cif_path) for cif_path in cif_paths]

for phase in phases:
    # use a smaller lattice range for each phase
    phase.params["lattice_range"] = 0.01


refinement = do_refinement_no_saving(
    data / pattern_fn,
    phases=phases,
    # if one parameter is both specified in phase_params and in the RefinementPhase object, the value in RefinementPhase will be used.
    phase_params={
        "lattice_range": 0.05,  # <- this will be ignored because it has already been set in the eahc RefinementPhase object
        "b1": "0_0^0.005",
        "k1": "0_0^1",
        "k2": "fixed",
        "gewicht": "SPHAR2"
    }
)

Refining with different instrument profiles, angle range, wavelength, etc.#

If you would like to do refinement in a different instrument profiles or angle range, you can specify it in the refinement function as well.

  • instrument_name: the instrument profile you would like to use. The default instrument profile used is Aeris-fds-Pixcel1d-Medipix3. You can find the available instrument profiles in the dara/data/BGMN-Templates/Devices folder.

  • wavelength: the wavelength you would like to use in the refinement. It can be two types:

    • a number: the wavelength in nm. It is useful when you analyzing the data from a synchrotron.

    • a string: the element symbol. It represents the target material in the X-ray tube. BGMN can automatically find the distribution of the wavelength for the given metal. Currently, it supports sources of [“Cu”, “Co”, “Cr”, “Fe”, “Mo”]

  • wmin, wmax: the angle range you would like to use in the refinement. It is set in refinement_params.

refinement = do_refinement_no_saving(
    data / pattern_fn,
    cif_paths,
    instrument_profile="Aeris-fds-Pixcel1d-Medipix3",
    wavelength="Cu",
    refinement_params={
        "wmin": 20,  # set the minimum two-thera in the refinement to be 20 deg.
        "wmax": 50  # set the maximum two-theta in the refinement to be 50 deg.
    }
)

refinement.visualize()

Save the project file to a folder on the disk#

Usually, you don’t have to directly interact with the BGMN input/output files.

However, if you would like to save the project directory for later use, you can use the do_refinement function instead.

The refinement files will be saved in the path specified by working_dir. Other than that, do_refinement and do_refinement_no_saving share the same parameters and output.

Feel free to modify the refinement project file yourself or by loading with the Profex software.

from dara import do_refinement

refinement = do_refinement(data / pattern_fn, cif_paths, working_dir="tutorial_refinement")

refinement_folder = Path("tutorial_refinement")

# show all the files in the folder
for file in refinement_folder.glob("*"):
    print(">", file.name)

> NiO_225_sym.str
> CaNi(PO3)4_800_240_Ca(OH)2_(NH4)2HPO4_NiO.xy
> CaNi(PO3)4_800_240_Ca(OH)2_(NH4)2HPO4_NiO.lst
> CaNi(PO3)4_800_240_Ca(OH)2_(NH4)2HPO4_NiO.par
> CaNi(PO3)4_800_240_Ca(OH)2_(NH4)2HPO4_NiO.sav
> CaNi(PO3)4_800_240_Ca(OH)2_(NH4)2HPO4_NiO.dia
> Aeris-fds-Pixcel1d-Medipix3.geq
> CaNi(PO3)4_15_sym.str